Default Bayesian analysis with global-local shrinkage priors
Extracted source text ยท arXiv 1510.03516
\documentclass[letterpaper,11pt]{article}
%\usepackage[slantedGreek]{mathpazo}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage[top=8pc,bottom=8pc,left=8pc,right=8pc]{geometry}
%\usepackage{ntheorem}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{subcaption}
\usepackage{multirow}
%% Please use the following statements for
%% managing the text and math fonts for your papers:
\usepackage{times}
%\usepackage[cmbold]{mathtime}
\usepackage{bm}
% this order is important
\RequirePackage[hyphens]{url}
\RequirePackage[colorlinks,citecolor=blue,urlcolor=blue]{hyperref}
\usepackage[authoryear]{natbib}
\usepackage[plain,noend]{algorithm2e}
%% For compressing some space:
\setlength{\textfloatsep}{10pt plus 1.0pt minus 2.0pt}
\setlength{\floatsep}{12.0pt plus 2.0pt minus 5.0pt}
\setlength{\intextsep}{12.0pt plus 2.0pt minus 5.0pt}
\setlength{\belowcaptionskip}{-2pt}
%\setlength{\textheight}{9in}
%\setlength{\textwidth}{6in}
%\setlength{\topmargin}{-36pt}
%\setlength{\oddsidemargin}{0pt}
%\setlength{\evensidemargin}{0pt}
\usepackage{array}
\newcolumntype{L}[1]{>{\raggedright\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcolumntype{R}[1]{>{\raggedleft\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}
\newcommand{\beginsupplement}{%
\setcounter{equation}{0}
\setcounter{page}{0}
\setcounter{table}{0}
\setcounter{section}{0}
\setcounter{figure}{0}
%\numberwithin{table}{section}
\renewcommand{\theequation}{S.\arabic{equation}}
\renewcommand{\thesection}{S.\arabic{section}}
\renewcommand{\thesubsection}{S.\arabic{section}.\arabic{subsection}}
\renewcommand{\thepage}{S.\arabic{page}}
\renewcommand{\thetable}{S.\arabic{table}}
\renewcommand{\thefigure}{S.\arabic{figure}}
}
\include{math-commands}
\renewcommand {\S}{\textsection} %%%%AB 4.29.16
\title{Default Bayesian analysis with global-local shrinkage priors}
\author{Anindya Bhadra \footnote{{\em Address:} 250 N. University St., West Lafayette, IN 47907, email: bhadra@purdue.edu.} \\Purdue University
\and Jyotishka Datta \footnote{{\em Address:} Box 90251, Durham, NC 27708, email: jd298@stat.duke.edu.}\\ Duke University and SAMSI\\
\and Nicholas G. Polson \footnote{{\em Address:} 5807 S. Woodlawn Ave., Chicago, IL 60637, email: ngp@chicagobooth.edu.} \ \ and Brandon Willard \footnote{{\em Address:} 5807 S. Woodlawn Ave., Chicago, IL 60637, email: bwillard@uchicago.edu.} \\The University of Chicago Booth School of Business}
\date{\vspace{-0.1cm} \today}
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\pagenumbering{arabic}
\begin{document}
\maketitle
\maketitle
\begin{abstract}
\noindent We provide a framework for assessing the default nature of a prior distribution using the property of regular variation, which we study for global-local shrinkage priors. In particular, we demonstrate the horseshoe priors, originally designed to handle sparsity, also possess regular variation and thus are appropriate for default Bayesian analysis. To illustrate our methodology, we solve a problem of non-informative priors due to \cite{efron1973discussion}, who showed standard flat non-informative priors in high-dimensional normal means model can be highly informative for nonlinear parameters of interest. We consider four such problems and show global-local shrinkage priors such as the horseshoe and horseshoe+ perform as \cite{efron1973discussion} requires in each case. We find the reason for this lies in the ability of the global-local shrinkage priors to separate a low-dimensional signal embedded in high-dimensional noise, even for nonlinear functions.
\end{abstract}
%\begin{keywords}
%Default Bayes; Global-local shrinkage; Non-informative prior; Regular variation; Robustness.
%\end{keywords}
\section{Introduction}\label{sec:intro}
\subsection{Default Bayesian analysis in multi-parameter problems}
In the discussion to \citet{dawid1973marginalization}, \citet{efron1973discussion} observed that the traditional default prior for the normal means could actually be highly informative for a low-dimensional nonlinear function of the means. \citet{efron1978controversies} further observes, ``the trouble with the multi-parameter estimation problem is not that it is harder than estimating a single parameter. It is easier, in the sense that dealing with many problems simultaneously can give extra information not otherwise available. The trouble lies in finding and using the extra information.'' Bayes' rule is clear on how to perform inference for a multivariate mean $ \theta=(\theta_1 , \ldots , \theta_p)$, as the posterior density is $ p( \theta \mid y ) \propto p( y \mid \theta ) p(\theta)$. The problem is in specifying $p(\theta)$.
Consider the classic normal means model $(y_i \mid \theta_i) \sim \Nor (\theta_i, 1)$ independently for $i = 1, \ldots, p$. A common statistical inference problem occurs when there is a high-dimensional parameter vector $\theta$ but the parameter of interest $\psi$ is a low-dimensional nonlinear function. In the absence of relevant prior information, \citet{Jeffreys1961} argues ``if we have no information relevant to the actual value of the parameter, the probability must be chosen so as to express the fact that we have none.'' Jeffreys' non-informative prior, $p(\theta) \propto |I(\theta)|^{1/2}$, where $I(\theta)$ is the Fisher information, is based on preservation of invariance under transformation, although for the normal means problem, he proposed a departure by using $\pi(\mu,\sigma) \propto 1/\sigma$. Unfortunately, Jeffreys' prior is often improper. \cite{dawid1973marginalization} argue that the posterior distributions for the improper priors have an unBayesian property, the result of the so-called marginalization paradox, which can be alleviated by using proper priors.
\cite{efron1973discussion} illustrates that when the parameter of interest is a nonlinear function of normal means, the posterior could lead away from a non-informative answer even when using a proper, and yet flat prior on $\theta$, and argues that the flat priors only make sense for group-transformation models. However, group transformation encompasses only a limited class of models, and in many cases the parameter of interest could be a complicated nonlinear quantity beyond the reach of the flat priors. \cite{bernardo1979reference} introduced the important notion of a parameter of interest where one re-parameterizes $ \theta=(\psi, \zeta)$ where $\psi$ is a parameter of interest and $\zeta$ is a vector of nuisance parameters that requires marginalization.
These re-parameterizations have little effect on simulation such as Markov chain Monte Carlo as one simply transforms the simulated draws from one parameterization to another. The issue that arises, however, is in specifying the default priors for inference. Transforming a prior, $p(\theta)$, from a high-dimensional space, to a low-dimensional, generally nonlinear parameter of interest, $p(\psi)$, involves a Jacobian or density transform that can take a seemingly non-informative prior into a highly informative one, leaving the researcher with a posterior distribution given the data that does not represent beliefs associated with non-informativity. Jeffreys' prior suffers directly from these issues and the growth of reference priors \citep{bernardo1979reference, berger1992development} was designed directly to handle the issue of non-informative inference for a parameter of interest while marginalizing out high-dimensional nuisance parameters. The problem of specifying a non-informative prior for a parameter of interest $ \psi $ then reduces to the properties of the marginal posterior of $\psi$ given $y$. The reference priors of \cite{bernardo1979reference} are constructed to not suffer from marginalization paradoxes. However, these priors are typically not integrable, which can lead to problems in more complicated settings such as hierarchical models \citep{gelman2006prior}. %See \citet{efron1973discussion} for a lively discussion of \citet{dawid1973marginalization} on the marginalization paradoxes. As part of the discussion of the latter paper,
\citet{efron1973discussion} posed a number of challenges for a Bayesian framework for default prior analysis, including the sum of squares and maximum of normal means. Specifically, he noted that priors such as independent $N(0,A)$ prior for each $\theta_i$ where $ A \rightarrow \infty$ can be highly informative for non-linearly transformed low-dimensional parameters of interest, such as, $\psi = \sum_{i=1}^{p} \theta_i^2$. A similar issue arises with estimating the variance $\sigma^2$ in the model $(y \mid \theta, \sigma^2) \sim \Nor (\theta, \sigma^2\I)$ under the inverse gamma prior $\sigma^2 \sim IG ( \epsilon, \epsilon ) $ as $ \epsilon \rightarrow 0$. \citet{gelman2006prior} proposed a $C^+(0,2{\cdot}5)$ prior for the scale parameter in a hierarchical model, where $\sigma \sim C^+(0,\tau)$ denotes a half-Cauchy distributed random variable, with density $p(\sigma \mid \tau) = 2/\{\pi \tau(1 + (\sigma/\tau)^2\}$, $\sigma > 0, \tau > 0$.
We argue here that the class of global-local shrinkage priors \citep{polson2012local} are good candidates for default priors. This class includes the horseshoe prior \citep{carvalho2010horseshoe} and the horseshoe+ prior introduced in an unpublished 2015 technical report by the authors of the current article. They are also particularly effective when the underlying parameter vector is sparse and the parameter of interest is a low-dimensional, possibly nonlinear function. \cite{gelman2006prior} already hinted at the default nature of half-Cauchy priors in hierarchical models. The question we consider is: what is the special property that distinguishes the half-Cauchy prior from other so-called flat priors? We will argue that the property of regular variation is key to a non-informative analysis. We study this property for the class of global-local shrinkage priors and show, via the use of dual densities, how to construct such priors. %An important feature of regular variation is closure under many nonlinear transformations such as sum of squares or maximum, with the appropriate transform of the index that preserves this property.
%Our view of being `non-informative' is one based on sensitivity analysis or one should let the data `speak for themselves' -- through the marginal likelihood $L(\psi) = \int L(\psi, \zeta) d \zeta $. Although, one should always plot prior versus posterior; since Bayes is a mechanism for `learning' or making relative, not absolute, judgments, between the prior and the likelihood.
While the use of heavy-tailed robust priors for non-informative inference has a long history \citep{dawid1973marginalization, o1979outliers,o1990outliers}, our new insight is that it is important to use sparse-robust priors as well, that is, ones that offer heavy shrinkage towards zero. For example, in the ratio of means, also known as the Fieller--Creasy problem \citep{fieller1940biological,creasy1954limits}, where $\psi= \theta_1 / \theta_2 $, one needs a prior that will let the data speak for themselves at the origin of $\theta_2$. In the case where the true parameter values are $ \theta_1 = \theta_2 = 0$, the likelihood ought to be uninformative and we should have a posterior with a similar spread as the prior.
%%% `Biometrika' papers do not contain a contents of the paper paragraph - per the Styleguide (introduction section).
%The rest of the paper is outlined as follows. \S \ref{sec:gl} reviews the necessary notation and details of global-local shrinkage priors with particular focus on horseshoe priors. \S \ref{sec:heavy} introduces the notion of regular variation and discusses its connection with prior sensitivity and construction of regularly varying priors via dual densities. \S \ref{sec:horseshoe} describes the four marginal parameters of interest problem we take up in this paper. In \S \ref{sec:results}, we numerically compare the performance of the global-local shrinkage priors with that of the other common `flat' priors and illustrate that the class of global-local class of priors performs satisfactorily in Efron's paradoxes. Thoeretical properties through posterior concentration are assessed in \S \ref{sec:post-conc}. Finally, \S \ref{sec:discuss} concludes with directions for future research.
\subsection{Global-local shrinkage} \label{sec:gl}
%\noindent \textbf{Global-local shrinkage}
We define the following class of priors on $\theta_i$ as global-local shrinkage priors
$$
(\theta_i \mid \lambda_i, \tau) \sim \Nor(0, \lambda_i^2), (\lambda_i \mid \tau) \sim p (\lambda_i \mid \tau), \tau \sim p(\tau), \quad \lambda_i > 0, \tau > 0,
$$
where $p(\lambda_i \mid \tau)$ has a tail that decays slower than an exponential rate.
%The horseshoe+ prior, as a scale mixture of normals, is given by
%\begin{align*}
% (\theta_i | \lambda_i, \tau) &\sim \Nor(0, \lambda_i^2) \\
% \lambda_i &\sim \operatorname{C}^+(0,\eta_i \tau) \\
% \eta_i &\sim \operatorname{C}^+(0,1)
%\end{align*}
%The original horseshoe is
%\begin{align*}
% (\theta_i | \lambda_i, \tau) &\sim \Nor(0, \lambda_i^2) \\
% \lambda_i &\sim \operatorname{C}^+(0,\tau)
%\end{align*}
Our approach to the marginal parameters of interest problems will be based on the use of global-local shrinkage priors \citep{polson2012local, polson2012half}. In particular, we focus on the horseshoe and horseshoe+ priors where the hierarchical models are specified as follows for the horseshoe,
\begin{align*}
% ( y_i | \theta_i ) &\sim N( \theta_i,1),\nonumber \\
( \theta_i \mid \lambda_i, \tau ) &\sim \Nor \left ( 0 , \lambda_i^2 \right ), ( \lambda_i \mid \tau ) \sim C^+ \left ( 0 , \tau \right ), \tau \sim C^+ \left ( 0 , 1 \right ), \lambda_i, \tau > 0,
%\\ \tau &\sim C^+ \left ( 0 , 1 \right ),
\end{align*}
and for the horseshoe+,
\begin{align*}
% ( y_i | \theta_i ) &\sim N( \theta_i,1), \nonumber \\
( \theta_i \mid \lambda_i, \eta_i, \tau ) &\sim \Nor \left(0 , \lambda_i^2 \right), (\lambda_i \mid \eta_i,\tau) \sim C^+ \left( 0 , \tau \eta_i \right), \\
\eta_i & \sim C^+ \left( 0 , 1 \right), \tau \sim C^+ \left ( 0 , 1 \right ), \lambda_i, \eta_i, \tau > 0.
\end{align*}
%Horseshoe priors do well for the problems of \citet{efron1973discussion} because they possess both tail-robustness and sparse-robustness properties.
A feature of the horseshoe and horseshoe+ priors is that they possess both tail-robustness and sparse-robustness properties, in other words, an infinite spike at the origin and as heavy a tail as possible that still ensures integrability.
%%
It is useful to view these priors by transforming to a shrinkage scale with
$\kappa_i = 1/(1 + \lambda_i^2 \tau^2)$. This transformation yields
the densities of global-local shrinkage priors,
$$
p(\theta_i \mid \tau) = \int_0^1 p(\theta_i \mid \kappa_i, \tau) p(\kappa_i \mid \tau) d\kappa_i,
\; p(\theta_i \mid \kappa_i, \tau) \sim \mathcal{N} \left(0, \frac{1-\kappa_i}{\kappa_i} \right).
$$
Here $ \kappa_i \in [0,1]$ is a component-specific shrinkage weight and $\tau$ is a global parameter.
In the case of a sparse $\theta$, only a few components are away from zero. \citet{west84} notes that the solution in this case is not to use a spherically symmetric heavy-tailed prior, such as a multivariate $t$. Indeed, for most of the components, the prior belief matches the observations, differing only in a few coordinates. Thus, the solution lies in putting component-specific priors on $\theta$. This is also the basic idea in global-local shrinkage priors. The global $\tau$ provides shrinkage towards zero for all components and indeed, the marginal prior on $\theta_i$ has unbounded spike at zero. The local, or component-specific $\lambda_i$ terms leave the large signals un-shrunk. The further half-Cauchy priors on $\lambda_i$ and $\tau$ allow these terms to be learned from the data. There are several optimality results available for the horseshoe and horseshoe+ priors \citep[][for the horseshoe; and a 2015 technical report by the current authors for the horseshoe+]{carvalho2010horseshoe,datta2013asymptotic,van2014horseshoe}. However, all of these results are concerned with the estimation of the multivariate normal mean itself, rather than its low-dimensional functions, which is the focus of the current work. We now discuss a key property that allows the global-local priors to perform well in the problems we consider: that of regular variation.
\section{Regular Variation and Non-informative Priors}\label{sec:heavy}
\subsection{Regular variation and Bayes' rule}\label{sec:rv-bayes}
We consider Bayes' rule for the parameter of interest $\psi$
$$
p( \psi \mid y ) = \frac{ L(\psi) p(\psi)}{p(y)}, \; \text{where} \; \; L(\psi) = \int p( y \mid \xi , \psi ) p( \xi \mid \psi ) d \xi,
$$
is the marginal likelihood. We need a prior $p(\theta_1, \ldots , \theta_p)$ that has a conditional $p(\xi \mid \psi) $ that leads to a well-behaved marginal likelihood and we need the prior $p(\psi)$ to have regular variation relative to this marginal likelihood. A positive, measurable function $f$ is called regularly varying at infinity with index $\alpha$ if $f$ is defined on some neighborhood $[x_0,\infty)$ and $\lim_{x \to \infty} f(tx)/f(x) \to t^{\alpha}$ for all $t>0$. The function $f$ is called slowly varying if $\alpha = 0$. A random variable $Y$ with distribution $F$ is defined to have a regularly varying upper tail if its survival function $\bar{F}$ is regularly varying at infinity. The regularly varying functions also satisfy a number of Tauberian theorems \citep{bingham1989regular}. A useful outcome of these theorems is that they are closed under most common many-to-one functions, for example, sums of regularly varying random variables are also regularly varying. The last property also characterizes a wider class of heavy-tailed distributions called sub-exponential distribution, which has also been linked to Bayesian robustness by \cite{andrade2011bayesian-b}. %See the Supplementary Material.
%Appendix~\ref{sec:regvar}
A key idea in Bayesian robustness modeling is the use of heavy-tailed distributions to resolve the conflict between different sources of information, namely the prior and outliers in data, that light-tailed distributions such as Gaussian fail to account for. The idea goes back to \citet{de1961bayesian}, \citet{lindley1968choice}, and \citet{dawid1973marginalization}. \citet{lindley1968choice} showed, by means of an approximate analysis, that if the likelihood is $y \mid \theta \sim \Nor(\theta, \sigma^2)$, then the posterior mean for $\theta$ given $y$ would not be too far from the observation $y$, if one uses a Student's $t$ prior instead of a Gaussian prior on $\theta$. \citet{dawid1973posterior} generalized the result for a pure location family and provided conditions that the prior ``should not wag its tail too vigorously," in order to achieve outlier rejection. The connection between heavy tails and Bayesian robustness has been explored in depth in the subsequent literature by \citet{o1979outliers,o1990outliers,andrade2006bayesian,andrade2011bayesian} and \citet{o2012bayesian}. In particular, \citet{andrade2006bayesian,andrade2011bayesian} established easily verifiable properties for conflict resolution through regular variation of the prior for location-scale families.
%In a parallel work, \citet{barndorff1982normal} deduced the relationship between tail behavior of a normal variance-mean mixture distribution and the tail behavior of its mixing distribution, leading to easy calculation of the marginal density for a wide variety of mixing distributions.
A remarkable result in the theory of regular variation is due to \cite{barndorff1982normal} and we show its use in Bayesian robustness. Their theorem shows that for any mixing distribution with slowly varying tail behavior, the corresponding normal variance mixture will also have slowly varying tails. In fact, the argument can be extended to any normal mean-variance mixtures by identifying a mean-variance mixture as an exponentially tilted version of the pure variance-mixture. This result shows that putting a slowly varying prior on the local shrinkage parameters $\Lambda = (\lambda_1^2, \ldots, \lambda_p^2)$ is a straightforward way of constructing slowly varying priors for the parameter of interest $\theta$.
\begin{theorem}\label{th:barndorff} \citep{barndorff1982normal}.
Let $p(\cdot)$ be a normal scale mixture $p(\theta) = \int (2\pi v)^{-1/2} e^{-\theta^2/2v} dF(v)$ and suppose that the mixing density $f(\cdot)$ satisfies
\beq
f(v) = e^{-\psi^+v} v^{\alpha-1} L(v), \; v \to \infty, \nonumber
\eeq
where $\psi^+ = \sup\{ s \in \Re: \phi(s) = \int e^{su} F(du) < \infty \}$, and $L(\cdot)$ is slowly-varying. Then,
\[
p(\theta) \sim
\begin{cases}
(2\pi)^{-\half} 2^{\half-\alpha} \Gamma(\half-\alpha) |\theta|^{2\alpha-1} L(\theta^2) , \; \theta \to \infty, \psi^+ = 0 \\
(2\psi^+)^{-\half \alpha} L(|\theta|) |\theta|^{\alpha-1} \exp\{-(2 \psi^+)|\theta|\}, \; \theta \to \infty, \psi^+ > 0.
\end{cases}
\]
\end{theorem}
The results of \citet{barndorff1982normal}, together with Tweedie's formula for the exponential family \citep{efron2011tweedie} can be used for checking posterior robustness for scale mixtures of normals. Results on regular variation are reviewed in the Supplementary Material. %Appendix~\ref{sec:regvar}
Unlike the half-Cauchy prior, the normal and Laplace prior do not work well for these problems as they lack regular variation. A more subtle problem occurs for the inverse-gamma $(\epsilon, \epsilon)$ prior for the variance term $\nu$. The inverse-gamma prior has regular variation at the right tail but not at the origin as the $\exp(-\epsilon/\nu)$ term dominates $\nu^{-(1+\epsilon)}$ term near the origin, which leads to shrinkage away from the origin.
We argue that the four classical one-dimensional problems introduced in \S \ref{sec:horseshoe} are examples of nice many-to-one functions that preserve the regular varying tails in the prior. It can be then argued that the heavy-tailed normal scale mixture priors such as the horseshoe and horseshoe+ will outperform priors for scale parameters with rapidly varying tails that make them unsuitable for robustness or conflict resolution.
%%
\subsection{Regular variation as a measure of flatness}
It has been argued that flat priors only make sense in group-transformation models \citep{efron1973discussion}. However, only a few models fall into that class, and often the goal is inference of complicated non-linear parameters of interest. As Bernardo points out in a conversation with \citet{irony1997non}, flat priors are usually very informative even when they are proper, and Jeffreys' prior works for regular, one-parameter problems but leads to marginalization paradoxes that both reference priors and the half-Cauchy priors seem to avoid for these multi-parameter problems. The heavy-tails of the half-Cauchy priors can be interpreted as a translation of vague prior beliefs \citep{rubio2014inference}. The usual non-informative priors become highly informative when one transforms a high-dimensional parameter into a low-dimensional parameter. %A distinct advantage of priors with regularly varying tails is that the regular variation property is closed for a wide variety of many-to-one functions. This helps to preserve the flatness of the priors even in the context of a marginal parameter of interest.
However, regular variation, due to closure properties under a wide variety of transformations, preserves the flatness of the priors even in the context of transformed parameters of interest.
%(see Appendix \ref{sec:regvar} for more details on regular variation)
\subsection{Regular variation and nonlinear functionals}
%As we have pointed out in \S \ref{sec:rv-bayes}, the regular variation property has been connected to Bayesian robustness by a number of authors, most notably Tony O'Hagan. Here we point out that these properties are crucial for solving the Efron paradoxes of flat priors. %It is interesting to note the solution to outliers by \cite{jaynes2003probability} and further emphasized in a recent unpublished article by Gelman and Hannig (2015). Jaynes suggested that the prior models have to specified for both ``good'' and ``bad'' data, including parameter priors for both models and prior probability for any number of observations to be bad. We find this to be in a similar spirit as the global-local shrinkage priors.
Assume $\tau=1$ without loss of generality. The global-local prior \citep{polson2012local} has the density of $\lambda_i$ of the form $p(\lambda_i^2) = K (\lambda_i^2)^{-a-1} L(\lambda_i)$ for some $a>0$. \cite{polson2010shrink} and \cite{ghosh2015asymptotic} show that such priors achieve tail-robustness as well as asymptotic Bayes optimality when $L(\cdot)$ is a slowly varying function. The marginal prior on $\theta_i$ is
\begin{equation*}
p(\theta_i ) \propto \int (2\pi\lambda_i^2)^{-1/2} \exp\left(-\frac{\theta_i^2}{2 \lambda_i^2}\right) \lambda_i^{-a-1} L(\lambda_i^2) d (\lambda_i^2), \lambda_i >0, a \geq 0.
\end{equation*}
An application of Theorem \ref{th:barndorff} gives
$$
p(\theta_i) \propto \left( \theta_i^2 \right)^{-a-1} L(\theta_i^2), |\theta_i | \to \infty,
$$
which shows $\theta_i$ has a polynomially decaying density and hence a regularly varying right tail. Recall that the likelihood is normal and we have $[y_i \mid \theta_i] \sim \Nor(\theta_i,1)$. Thus, Result 2 of \citet{dawid1973posterior} implies $ [\theta_i \mid y_i] \rightsquigarrow \Nor(y_i, 1) $ as $|y_i | \to \infty$.
Hence the posterior of $\theta_i$ corresponding to a large $y_i$ is centered at the observation $y_i$, with the same standard deviation as the data distribution. Thus, the posterior estimate of such a $\theta_i$ will be the same as pure likelihood-based estimate and prior information is discounted. If one defines low dimensional functions of $\theta$ such as $\psi = \sum_{i=1}^{p } \theta_i^2, \psi = \max \theta_i, \psi = \theta_1\theta_2, \psi = \theta_1/\theta_2$, by closure properties of regular variation, the prior $p(\psi)$ will have regular variation. Thus the posterior estimates of $\psi$ will also be the same as the likelihood-based estimates, provided corresponding $y_i$s are all large. For a normal prior, the prior and the likelihood have the same tail heaviness, and the results of \citet{andrade2006bayesian} imply that the posterior of $(\theta_i \mid y_i)$ will not be the same as the likelihood-based estimate. Hence, the resultant posterior estimates of $\psi$ will differ from their likelihood-based estimates, explaining Efron's observations.
From this description, it is natural to wonder if just putting a heavy-tailed prior such as a multivariate-$t$ on $\theta$ will suffice. However, in a sparse true $\theta$ case, most of the observed small $y_i$s are just noise terms. There is another key property of global-local priors that distinguishes them from other priors with heavy tails, such as a multivariate-$t$. Global-local priors also offer heavy shrinkage towards zero. Specifically, the density $p(\theta_i)$ has an unbounded spike near zero and the global shrinkage parameters adapts itself to the sparsity level of the data, when the true parameter vector is sparse. The unbounded spike near zero for horseshoe and horseshoe+ are respectively demonstrated by \citet{carvalho2010horseshoe} and in a technical report by the current authors. This is not the case with a multivariate-$t$ prior on $\theta$ and the terms with small $y_i$ are not shrunk as effectively. In presence of a true $\theta$ that is sparse-robust, that is, most components are zero and a few components are large, the heavy tails of the marginal prior on $\theta_i$ in combination with its unbounded spike near zero explains why global-local shrinkage priors perform so effectively.
%\begin{enumerate}
%\item Add intuition for Dawid's proof
%\item Repeat bernardo parameter of interest framework.
%\item {\color{red} Add a subsection to distinguish from O'Hagan's framework. Why is global shrinkage needed? Just RV isn't enough.}
%\end{enumerate}
\subsection{Duality of densities}
Since the regularly varying global-local shrinkage priors arise as normal scale mixtures, it is natural to ask what properties of the mixing distribution ensures a regularly varying right tail of the shrinkage prior, and how to generate new regularly varying shrinkage priors that are also normal scale mixtures.
Suppose that $p(x) = \int (2\pi v)^{-1/2} e^{-x^2/2v} dF(v)$, is a normal scale mixture with moment generating function $\phi$. Theorem \ref{th:barndorff} can be used to establish the relationship between the tail behavior of the mixing distribution and the tail behavior of the normal scale mixture.
Turning to the second question, an important tool for constructing new regularly varying priors is by using the dual densities for known normal scale mixtures. The concept of dual densities was introduced by \citet{good1995c439} as densities proportional to the characteristic functions (or moment generating functions when they exist). Specifically, we construct
$$
\hat{p} (t) = \frac{ \phi(t)}{C}, \; \; \phi(t) = E \{\mathrm{exp}(tX)\}, \; C = \int_{\mathcal{D}} \phi(y) dy,
$$
where $\mathcal{D}$ is such that $\hat{p}(t) \geq 0$ for all $t \in \mathcal{D}$ and $\int_{\mathcal{D}} \phi(y) dy < \infty$.
This dual density $\hat{p}$ is also a normal scale mixture with mixing density $\hat{f}$. The mixing density for the dual and the original density denoted by $\hat{f}$ and $f$ are related by:
$$
\hat{f}(v) = \frac{1}{(2\pi)^{1/2}p(0)v^{3/2}} f \left(\frac{1}{v}\right) , \; v > 0.
$$
The purpose of this is to be able to construct regularly varying priors that are also normal scale mixtures starting from rapidly varying priors. Dual densities were further explored by \citet{gneiting1997normal} and \citet{nadarajah2009pdfs}. See Table \ref{tab:rv-densities} for a taxonomy of dual densities which can be used to construct slowly varying global-local scale mixtures. The classic example is the double-exponential or Laplace distribution whose dual is the Cauchy distribution. Since the Laplace can be written as a normal scale mixture with exponential mixing density, we can derive the mixing density for Cauchy.
\begin{table}[ht!]%
\centering
\def~{\hphantom{0}}
{\footnotesize
\caption{Some common exponentially tailed densities that have polynomially tailed dual densities. The densities marked with asterisks also have a known normal scale mixture representation.\label{tab:rv-densities}}
\begin{tabular}{ L{35mm} L{45mm} R{45mm} }
Density $p(x)$ & Dual Density $\hat{p}(x)$ & Comments \\
\hline
Exponential Power* (Special Case: Laplace, Normal) & Symmetric-stable($\alpha$) (Special case: Cauchy, Levy) & Symmetric stable distributions are heavy-tailed for $\alpha<2$. \\
\hline
Bessel function density* & Student's t & These densities are special cases of Generalized Hyperbolic distribution with parameters ($\lambda=-\frac{\nu+1}{2},\delta^2=0,\kappa^2=\nu$) and ($\lambda=-\nu/2,\delta^2=\nu,\kappa=0$) respectively. \\
\hline
Gamma (shape=$\alpha$,rate=$\lambda$) & $\hat{p}(x) = x^{-\alpha}$ & \\
\hline
Laplace* \newline Skew-Laplace I \newline Skew-Laplace II & $\hat{p}(x) \propto e^{ax}(1+x^2)^{-1}$ \newline $\hat{p}(x) \propto e^{ax}(c/(b+x)+1/x)$ \newline $\hat{p}(x) \propto \{\frac{cx}{x^2+a^2} + \frac{1}{b+x}\}$ & Heavy-tailed if $a=0.$ \newline Heavy-tailed if $a=0.$ \newline Heavy-tailed. \\
\hline
Frech\'et & $\hat{p}(x) \propto \sqrt{x}K_{-1}(a\sqrt{x})$ & Frech\'et distribution has a lower exponential tail. \\
\hline
Inverse Gaussian & $\hat{p}(x) \propto x^{1/4}K_{-\half}(a\sqrt{x})$ & \\
\hline
Linnik or $\alpha$-Laplace distribution* \newline $\alpha \in [0,2)$ & Generalized Cauchy \newline $\hat{p}(x) \propto (1+|x|^{\alpha})^{-\beta}$ & \\
\hline
\end{tabular}
}
\end{table}
%%%
\section{Examples of Nonlinear functions of multivariate normal means}\label{sec:horseshoe}
% \noindent We will apply the horseshoe and horseshoe+ priors to four examples, as detailed below.
\subsection{Example 1: sum of squares} \citet{efron1973discussion} shows the following example: if $p=100, \psi= \sum_{i=1}^{100} \theta_i^2 $ and we observe $ \sum_{i=1}^{100} y_i^2 = 200 $, then the estimate $\hat{\psi} = 100 $ with standard deviation $25$ is the minimum variance unbiased estimator. It is easy to see that the posterior mean under an uninformative prior $N(0,\tau^2)$, with a large $\tau^2$, is $300$, which is badly biased upwards. This is because the marginal prior on $\psi$ heavily shrinks $\psi$ away from the origin. To be concrete, observe that
$$
\psi = \sum_{i=1}^p \theta_i^2 \sim \Gamma \left ( \frac{p}{2} , \frac{\tau^2}{2} \right ), p(\psi) = \tau^{-p} \psi^{ \frac{p}{2} -1 } e^{ - \frac{\psi}{2 \tau^2} }.
$$
Therefore a large $\tau^2$ shrinks away from the origin due to the term $ \psi^{49}$ when $p=100$. One way of correcting this is to give $\tau$ a diffuse prior such as the half-Cauchy prior. Then the marginal prior is $p(\tau) \propto (\tau^2+1)^{-1}$ making the marginal for $\psi$ proportional to $ \psi^{-1/2} ( \psi + p )^{-1} $ and the posterior mean of $\psi$ close to the maximum likelihood estimate. To see this, we start with the marginal prior on $\psi$, which in this case is:
$$
p( \psi) = \int_0^\infty \tau^{-p} \psi^{ \frac{p}{2} -1 } e^{ - \frac{\psi}{2 \tau^2} } p(\tau^2) d \tau.
$$
As we show in Theorem \ref{th:post-ss}, the term $ p( \psi|\tau^2)$ collapses on $\tau^2 = \psi/p$ so we have a marginal prior on $\psi$ equal to the prior on $\tau^2$ evaluated at $\psi/p$. Hence we have
$$
p( \psi ) \propto \psi^{-1/2} ( \psi + p )^{-1}.
$$
This shrinks towards the origin, as the density is not zero there. This is the sense in which estimating or learning the hyperparameters helps in a non-informative analysis. \citet{berger1998estimation} points out the sub-optimality of the constant prior on $\theta$ which is also the Jeffreys' prior and derives the naive reference prior for estimating $\psi$, given by $p(\psi) = \psi^{-(p-1)/2}$. \citet{berger1999} point out the failure of profile likelihood approaches for this problem as it fails to account for the uncertainty in the nuisance parameter and show how the integrated likelihood approach avoids the pitfalls of marginalization paradox. We compare the reference prior with the other shrinkage priors and argue that the reference prior in this setting acts as a global shrinkage prior and works well in the dense case where $\theta_i = 1$ for all $i$, but its performance deteriorates in the sparse case due to over-shrinkage of large observations.
%However, there are difficulties with this solution.
\subsection{Example 2: maximum of normal means} This example, also due to \citet{efron1973discussion}, concerns estimating $ \psi = \max \theta_i$ given observations $y_1, \cdots , y_{99}$ drawn from standard normal distribution while $y_{100}=10$. Suppose in order to correct the problem with the sum of squares estimation under a fixed, large $\tau^2$, we have now used a prior $p(\tau^2) \propto (\tau^2 + 1)^{-2}$, as in Example 1. The posterior mean for $\psi$ is now $5$, while it is obvious that $10$ is a much more sensible estimate in this case. Now the problem arises because there is only one global shrinkage parameter $\tau$ with a heavy-tailed prior that shrinks the true signals as well as the noise terms. The James-Stein shrinkage estimator, being a purely global shrinkage estimator, suffers from the same problem. We will show that the situation can be improved by adding local shrinkage parameters in addition to a global $\tau$.
\subsection{Example 3: product of two normal means} The third marginal parameter inference problem deals with the product of normal means $\psi= \theta_1 \theta_2$ where we observe $(y_1, y_2 \mid \theta_1, \theta_2)' \sim \Nor((\theta_1,\theta_2)', \mathrm{I})$ \citep{berger1989estimating}. It follows that the reference prior in this context is proportional to $ ( \theta_1^2 + \theta_2^2 )^{-1/2}$, which has polynomially decaying tails like the half-Cauchy prior, but is not integrable. For a detailed review of reference prior for product normal means and its generalization to the multivariate case, we refer the readers to \citet{sun1995reference, sun1999reference}. We will show when both $\theta_1$ and $\theta_2$ are close to zero, the posterior distribution under the global-local priors concentrates around the origin, unlike its competitors, including normal, pure global and pure local shrinkage priors.
%It turns out that the half-Cauchy prior indeed do well for this problem (see \S \ref{sec:pm-fc}). \\
\subsection{Example 4: ratio of two normal means} Finally, we consider the Fieller--Creasy problem, where the parameter of interest is $\psi= \theta_1/\theta_2$ \citep{fieller1940biological,creasy1954limits}. Defining $\lambda = \theta_2$ as the nuisance parameter, the reference prior is obtained as $ p(\psi, \lambda) \propto (\psi^2 +1 )^{-1/2} $ \citep{liseo1993elimination} which is Cauchy-like but not integrable. One expects horseshoe to produce similar results and as we show in \S \ref{sec:pm-fc}, these priors lead to sensible answers by concentrating around the true value for $\theta_2 \neq 0$ and around the origin when $\theta_1 = \theta_2 = 0$. In all the examples, global-local priors enjoy the benefit of being integrable.
%%%% Simulation results section goes here %%%%
%%%%%%%%%%%%%%%%%%% Simulation Studies \S %%%%%%%%%%%%%%%%%%%%%%
%\section{Results}\label{sec:simulation}
\section{Results}\label{sec:results}
\subsection{Simulation scheme} We compare the following priors for the marginal parameter of interest problems of \S \ref{sec:horseshoe}.
\ben
\item The global-local shrinkage priors, namely, the horseshoe and the horseshoe+ priors. %
%The horseshoe+ prior, as a scale mixture of normals, is given by
%\begin{align*}
%(\theta_i | \lambda_i, \tau) &\sim \operatorname{N}(0, \lambda_i^2), \\
%(\lambda_i | \eta_i, \tau) &\sim \operatorname{C}^+(0,\eta_i \tau), \\
%\eta_i \sim \operatorname{C}^+(0,1), & \; \tau \sim \operatorname{C}^+(0,1).
%\end{align*}
%The original horseshoe prior is given by:
%\begin{align*}
%(\theta_i | \lambda_i, \tau) &\sim \operatorname{N}(0, \lambda_i^2), \\
%(\lambda_i | \tau) &\sim \operatorname{C}^+(0,\tau), \\
%\tau & \sim \operatorname{C}^+(0,1).
%\end{align*}
\item The Laplace or double-exponential prior, with the mixing density given by:
\begin{align}
p(\lambda_i^2 \mid \tau^2) & = (2\tau^2)^{-1}{\exp}\{ - \lambda_i^2/2\tau^2 \}, \label{eq:laplace-1} \\
(\tau^2 \mid \xi, d) & \sim {\mathrm IG}(\xi/2, \xi d^2/2), \xi >0, d > 0. \label{eq:laplace-2}
\end{align}
The Laplace prior has a long history in Bayesian robustness modeling \citep{pericchi1992exact, box1962further}. For the numerical examples, we use the standard double-exponential prior, that is, fix $\xi = d = 1$.
\item The vague normal prior, that is, $\theta_i \sim \Nor(0, \sigma^2 = 300)$.
\item The pure-local shrinkage prior, and the pure-global shrinkage priors, by taking $\tau = 1$ or, $\lambda_i = 1$, for all $i = 1, \ldots, p$, in the hierarchical model for horseshoe prior in \S 1.2. These two priors are included to highlight the role of both local and global shrinkage parameters.
\item The reference priors for Examples 1, 3 and 4.
\een
In the following simulations, the estimates are produced using the probabilistic programming language \texttt{Stan} \citep{stan-software:2014} according to the parametrization shown above. \texttt{Stan} is run with 4 parallel chains of 10,000 iterations. We also provide two efficient algorithms for sampling from the horseshoe for horseshoe+ posteriors in the Supplementary Material. %Appendix~\ref{sec:slice} and \ref{sec:trunc}
%We demonstrate and compare the horseshoe+ prior with the horseshoe for the problems mentioned in \citet{efron1973discussion} involving
%the estimation of many-to-one functions of $\theta_i$, say, $\psi_1(\btheta)$ and $\psi_2(\btheta)$ with:
%\begin{align}
%y_i & \sim \Norm\left(\theta_i, 1\right) \; i = 1, \ldots, n \label{eq:eq:efron_norm_obs} \\
%\psi_1(\btheta) & = \sum_{i=1}^{n} \theta_i^2; \; \psi_2(\btheta) = \theta_{(n)} =\max_{i = 1, \ldots,n} \{\theta _1, \ldots ,\theta_n \}
%\end{align}
\subsection{Sum of squares estimation}\label{sec:sum}
For comparing the candidate priors, we chose $\theta$ with $q_p$ non-zero components of magnitude $A$ and dimension $p = 100$. The simulation study shows the performance for different $\theta$ in both the sparse regime where $q_p/p \to 0$ and completely dense where $q_p/p \to 1$. Table \ref{tab:efron-ss} shows the performance comparison for the candidate priors for three different sparsity patterns while constraining $\theta$ to have unity quadratic mean, that is, $\vectornorm{\theta}^2 = p$. The chosen values for $A$ and $q_p$ are $A = 10, q_p = 1$, $A = 5, q_p = 4$, and $A = 1, q_p = 100$ respectively. The posterior quantiles, mean and standard deviation for $\psi = \sum_{i=1}^{p} \theta_i^2$ for the three experiments are given in Table \ref{tab:efron-ss}, and Figure \ref{fig:efron-ss} shows the corresponding posterior distribution of $\psi$.
The first two blocks of Table \ref{tab:efron-ss} and the sub-figures Fig. \ref{fig:efron-ss-sparse-1} and Fig. \ref{fig:efron-ss-sparse-2} shows the relative performance of the different priors in the two sparse situations, that is, $A = 10, q_p = 1$, and $A = 5, q_p = 4$. As expected, both horseshoe and horseshoe+ gives sensible answers in the sparse case with the posterior distribution centered near $100$ and standard deviation close to $25$ as desired by \citet{efron1973discussion}. The Laplace prior also does well for the sparse $\theta$, but as we shall see shortly, its performance deteriorates for the remaining examples, maximum, product of means and the Fieller--Creasy problem, advocating against its use as a default prior. The density plots Fig. \ref{fig:efron-ss-sparse-1} and Fig. \ref{fig:efron-ss-sparse-2} are instructive to see where the badly performing priors concentrate. The vague normal prior leads to an upward bias in all cases, as \cite{efron1973discussion} had pointed out. Interestingly, the pure-local shrinkage prior concentrates above as it can not adapt to the sparsity in absence of a global shrinkage parameter. On the other hand, pure-global prior and the reference prior shrinks the posterior more towards zero and lead to under-estimation, as one should expect when the only shrinkage mechanism is a global parameter.
The dense case is $\theta_i = 1$, for all $ i = 1, \ldots, p$. The global-local shrinkage priors, that clearly outperform the others in the sparse case, are not the best performers here as they are designed to shrink the small $\theta_i$s, which under-estimates $\sum_{i=1}^{p} \theta_i^2$. The pure-global shrinkage prior is the best performer in the dense case, followed closely by the reference prior. The global prior leads to a posterior that concentrates on $(0, \sum_{i=1}^{p} y_i^2)$, unlike the vague normal prior that concentrates above this region. We prove this theoretically in \S \ref{sec:post-conc}.
To summarize, the global-local shrinkage priors give a more sensible answer than the proper flat normal prior with a high variance in the sparse case. The normal prior not only gives a posterior mean far away from the true value, as Fig. \ref{fig:efron-ss} shows, the posterior for a vague normal prior puts most of its density on a different region, unlike the global-local shrinkage priors that concentrates near the true $\psi$. Even in the dense case, where the global-local shrinkage priors shrink too strongly, a global shrinkage prior with a half-Cauchy prior produces an accurate estimates, but one cannot trade the half-Cauchy with a $\Nor(0, 300)$ prior, suggesting regular variation as a key property for Bayesian robustness.
%The posterior concentration can be further understood from the last figure (Fig. \ref{fig:efron_sum_smpls_theta_plot}, where we show the histogram of draws from the posterior distribution of the marginal parameter $\psi = \sum \theta_i^2$ that lie in a middle 40\% region, under each of the three priors for the most sparse example above. As before, we can see that the posterior contracts nicely around the true value for horseshoe and horseshoe+ but drifts away for the vague normal prior as expected.
% Table showing six different scenarios%
\begin{table}[ht!]
\centering
\caption{Summary statistics and standard deviation of the posterior distribution for the candidate priors, namely, the horseshoe+ (HS+), the horseshoe (HS), Laplace, normal, pure-local, pure-global and the reference priors, for parameter of interest $\psi = \sum_{i=1}^{100} \theta_i^2$. Here $q_p$ is the number of non-zero means and $A$ is the magnitude of the non-zero means. True $\psi = 100$ in all cases.}% The winners in each setting are indicated by bold-faced entries.}
{\footnotesize
% Table generated by Excel2LaTeX from sheet 'SS'
\begin{tabular}{rrrrrrrrr}
& & Minimum & $Q_1$ & Median & Mean & $Q_3$ & Max & Std. Dev. \\
A = 10, $q_p = 1$ & HS+ & 46${\cdot}$7 & 84${\cdot}$4 & {97${\cdot}$6} & {98${\cdot}$3} & 111${\cdot}$6 & 172${\cdot}$8 & 19${\cdot}$8 \\
& HS & 46${\cdot}$6 & 82${\cdot}$3 & {94${\cdot}$6} & {96${\cdot}$6} & 108${\cdot}$7 & 159${\cdot}$0 & 19${\cdot}$2 \\
& Laplace & 44${\cdot}$9 & 86${\cdot}$7 & {101${\cdot}$7} & {103${\cdot}$5} & 119${\cdot}$3 & 180${\cdot}$4 & 23${\cdot}$2 \\
& Normal & 189${\cdot}$7 & 260${\cdot}$0 & 276${\cdot}$9 & 278${\cdot}$0 & 295${\cdot}$3 & 373${\cdot}$9 & 29${\cdot}$4 \\
& Local & 89${\cdot}$5 & 137${\cdot}$8 & 152${\cdot}$4 & 153${\cdot}$9 & 169${\cdot}$2 & 229${\cdot}$7 & 22${\cdot}$7 \\
& Global & 19${\cdot}$5 & 64${\cdot}$6 & 78${\cdot}$9 & 79${\cdot}$9 & 93${\cdot}$1 & 164${\cdot}$1 & 21${\cdot}$6 \\
& Reference & 29${\cdot}$4 & 66${\cdot}$3 & 81${\cdot}$1 & 82${\cdot}$5 & 98${\cdot}$0 & 162${\cdot}$8 & 23${\cdot}$9 \\
& & & & & & & & \\
& & Minimum & $Q_1$ & Median & Mean & $Q_3$ & Max & Std${\cdot}$ Dev${\cdot}$ \\
A = 5, $q_p = 4$ & HS+ & 53${\cdot}$2 & 89${\cdot}$8 & {104${\cdot}$0} & {105${\cdot}$9} & 121${\cdot}$4 & 182${\cdot}$8 & 22${\cdot}$8 \\
& HS & 49${\cdot}$8 & 88${\cdot}$2 & {103${\cdot}$1} & {103${\cdot}$6} & 119${\cdot}$0 & 180${\cdot}$3 & 21${\cdot}$6 \\
& Laplace & 50${\cdot}$6 & 87${\cdot}$9 & {104${\cdot}$0} & {105${\cdot}$0} & 119${\cdot}$2 & 214${\cdot}$2 & 23${\cdot}$8 \\
& Normal & 191${\cdot}$6 & 265${\cdot}$5 & 285${\cdot}$3 & 285${\cdot}$7 & 305${\cdot}$2 & 380${\cdot}$9 & 30${\cdot}$8 \\
& Local & 88${\cdot}$9 & 134${\cdot}$4 & 149${\cdot}$9 & 150${\cdot}$3 & 164${\cdot}$7 & 240${\cdot}$0 & 23${\cdot}$5 \\
& Global & 27${\cdot}$6 & 70${\cdot}$1 & 84${\cdot}$0 & 86${\cdot}$2 & 102${\cdot}$5 & 170${\cdot}$6 & 24${\cdot}$8 \\
& Reference & 34${\cdot}$36 & 69${\cdot}$64 & 85${\cdot}$3 & 86${\cdot}$5 & 100${\cdot}$9 & 164${\cdot}$7 & 23${\cdot}$7 \\
& & & & & & & & \\
& & Minimum & $Q_1$ & Median & Mean & $Q_3$ & Max & Std${\cdot}$ Dev${\cdot}$ \\
A = 1 , $q_p = 100$ & HS+ & 31${\cdot}$9 & 67${\cdot}$8 & 83${\cdot}$8 & 86${\cdot}$0 & 101${\cdot}$8 & 165${\cdot}$6 & 24${\cdot}$1 \\
& HS & 36${\cdot}$3 & 73${\cdot}$8 & {88${\cdot}$5} & {91${\cdot}$6} & 107${\cdot}$2 & 190${\cdot}$0 & 24${\cdot}$0 \\
& Laplace & 51${\cdot}$2 & 89${\cdot}$6 & {107${\cdot}$1} & {107${\cdot}$6} & 123${\cdot}$7 & 181${\cdot}$9 & 26${\cdot}$3 \\
& Normal & 219${\cdot}$3 & 288${\cdot}$0 & 308${\cdot}$8 & 309${\cdot}$0 & 331${\cdot}$4 & 442${\cdot}$6 & 32${\cdot}$5 \\
& Local & 76${\cdot}$4 & 123${\cdot}$7 & 138${\cdot}$0 & 138${\cdot}$4 & 153${\cdot}$2 & 216${\cdot}$3 & 23${\cdot}$3 \\
& Global & 53${\cdot}$23 & 94${\cdot}$73 & {108${\cdot}$0} & {110${\cdot}$2} & 125${\cdot}$7 & 198 & 22${\cdot}$9 \\
& Reference & 47${\cdot}$67 & 92${\cdot}$45 & 114${\cdot}$5 & 113${\cdot}$8 & 131${\cdot}$5 & 188${\cdot}$3 & 27${\cdot}$8 \\
\end{tabular}%
}
\label{tab:efron-ss}%
\end{table}%
%\figurebox{12.5pc}{15pc}{}[prior_diff_ab.eps]
%
\begin{figure}[ht!]
\centering
\begin{subfigure}[b]{0.45\textwidth}%
\raggedleft
\includegraphics[width=\linewidth]{re-revised-hist-efron-ss-1-10}%
\caption{$A = 10$ and $q_p = 1$.}%
\label{fig:efron-ss-sparse-1}%
\end{subfigure}
\begin{subfigure}[b]{0.45\textwidth}%
\raggedright
\includegraphics[width=\linewidth]{re-revised-hist-efron-ss-4-5.eps}%
\caption{$A = 5$ and $q_p = 4$ .}%
\label{fig:efron-ss-sparse-2}%
\end{subfigure}
\begin{subfigure}[b]{0.45\textwidth}%
\centering
\includegraphics[width=\linewidth]{re-revised-hist-efron-ss-100-1.eps}%
\caption{$A = 1$ and $q_p = 100$.}%
\label{fig:efron-ss-sparse-3}%
\end{subfigure}
\caption{Posterior densities of the parameter of interest $\psi = \sum_{i=1}^{100} \theta_i^2$, for the candidate priors, namely, the horseshoe+ (HSplus), the horseshoe (HS), Laplace, normal, pure-local, pure-global and the reference priors. In each panel, $q_p$ is the number of non-zero means and $A$ is the magnitude of the non-zero means. The horizontal line is at the true value $\psi = 100$. }
\label{fig:efron-ss}
\end{figure}
%
%%-----------------------------------------------------------%%
%% NOT SURE IF COMMENT THIS OUT BECAUSE IN MY SIMULATION HS, HS+ WAS DOING AS WELL AS NORMAL %%%
%% THEY WERE ALL GIVING NICE, SENSIBLE ANSWER AS WE EXPECT %%
%% --------------------------------------------------------- %%
\subsection{Maximum estimation}\label{sec:max}
Here we show that the horseshoe priors do well for estimating $\theta_{(p)}$, the maximum element of $\theta$, when one $y_i$ is an outlier at $10$ and the rest of the $y_i$s are standard normal variates. We expect that the global-local shrinkage priors to do well due to the local shrinkage parameters that provide robustness in the tails, that is $E(\theta \mid y) \approx y$ for large $|y|$, whereas a pure-global shrinkage prior results in a posterior mean of around $5$, as pointed out by \citet{efron1973discussion}. Indeed, as Table \ref{tab:efron-max} and Fig. \ref{fig:efron-max} show, both the global-local priors as well as the normal prior with high standard deviation do well for this example while the worst-performing candidate is the global shrinkage prior. Interestingly, the Laplace prior performs poorly due to its over-shrinkage on the tails. This agrees with fact that the Laplace prior leads to a biased estimate for large observations, as noted in the discussion in \S 3.2 of \cite{carvalho2010horseshoe}, where they showed that for large $|y_i|$ there is a constant bias, $|E(\theta_i | y_i) - y_i | \approx \sqrt{2}/\tau$, for a given $\tau$.
\begin{table}[ht!]
\centering
\caption{Summary statistics and standard deviation of the posterior distribution for parameter of interest $\psi = \max \theta_i$, for the candidate priors, namely, the horseshoe+ (HS+), the horseshoe (HS), Laplace, normal, pure-local and pure-global priors, for $[y_i \mid \theta_i] \sim \Nor(0,1)$ for $i = 1, \ldots, 99$ and $y_{100} = 10$.}
{\footnotesize
\begin{tabular}{rrrrrrrrr}
& & Minimum & $Q_1$ & Median & Mean & $Q_3$ & Max & Std.Dev. \\
& HS+ & 6$\cdot$7 & 9$\cdot$0 & 9$\cdot$7 & 9$\cdot$7 & 10$\cdot$4 & 12$\cdot$5 & 1$\cdot$1 \\
& HS & 6$\cdot$8 & 9$\cdot$2 & 9$\cdot$8 & 9$\cdot$9 & 10$\cdot$5 & 12$\cdot$9 & 1$\cdot$0 \\
& Laplace & 4$\cdot$8 & 7$\cdot$8 & 8$\cdot$5 & 8$\cdot$4 & 9$\cdot$1 & 11$\cdot$0 & 1$\cdot$0 \\
& Normal & 6$\cdot$6 & 9$\cdot$3 & 10$\cdot$0 & 9$\cdot$9 & 10$\cdot$6 & 13$\cdot$1 & 1$\cdot$0 \\
& Local & 6$\cdot$7 & 9$\cdot$1 & 9$\cdot$9 & 9$\cdot$8 & 10$\cdot$5 & 13$\cdot$0 & 1$\cdot$0 \\
& Global & 1$\cdot$5 & 3$\cdot$7 & 4$\cdot$4 & 4$\cdot$4 & 5$\cdot$1 & 7$\cdot$7 & 1$\cdot$0 \\
\end{tabular}%
}
\label{tab:efron-max}%
\end{table}%
\begin{figure}[ht!]%
\centering
\includegraphics[width=0.5\linewidth,height = 2in]{re-revised-hist-efron-max-1.eps}%
\caption{Posterior densities of the parameter of interest $\psi = \max \theta_i$, for the candidate priors, namely, the horseshoe+ (HSplus), the horseshoe (HS), Laplace, normal, pure-local and pure-global priors, for $(y_i \mid \theta_i) \sim \Nor(0,1)$, $i = 1, \ldots, 99$ and $y_{100} = 10$. The horizontal line is at $y_{\max}$.}%
\label{fig:efron-max}%
\end{figure}
\subsection{Bivariate normal means: product and ratio}\label{sec:pm-fc}
We compare the different shrinkage priors for Examples 3 and 4 in \S \ref{sec:horseshoe}, concerning the product and ratio of means \citep{berger1989estimating, creasy1954limits} of a bivariate normal distribution. The data are generated as $K = 100$ independent draws from a bivariate normal distribution as
\begin{align*}
(y_k \mid \theta_1, \theta_2) &\sim \Norm \left((\theta_1, \theta_2), \mathrm{I} \right), k = 1, \ldots, K.
\end{align*}
The sample mean for the data is $(\bar{y}_1, \bar{y}_2) = (0{\cdot}0427, -0{\cdot}0840)$. The results below pertain to the case where the true values for both $\theta_1$ and $\theta_2$ are zero. %As in \S \ref{sec:sum} and \S \ref{sec:max}, we consider the following priors: horseshoe, horseshoe+, double exponential using the hierarchical model in \eqref{eq:laplace-1} and \eqref{eq:laplace-2}, the pure-local and pure-global shrinkage priors, independent $\Nor(0,1)$ prior on each $\theta_i$ as well as the reference priors.
Figure~\ref{fig:fc_plot_theta_density} shows the two-dimensional density contours for all the candidate posteriors. Fig.~\ref{fig:fc_plot_theta_density} clearly illustrates that the posterior distributions under the horseshoe and the horseshoe+ priors are highly concentrated in a small interval near the origin unlike their competitors, such as the double-exponential, the normal, or the pure-local and pure-global shrinkage priors. As a result, the global-local shrinkage priors enjoy higher estimation accuracy compared to the competitors. The global shrinkage estimator also concentrates near the origin but with a higher spread, implying regular-variation is a key property for learning marginal parameters of interest. %{\color{red} (Reference prior for $\psi=\theta_1/\theta_2$ does the best in the table, but not in the figure?)}
\begin{figure}[ht!]%
\centering
\includegraphics[width=0.8\linewidth,height=2.5in]{8-contour-plots-grayscale-legend.eps}%
\caption{Two-dimensional contour plots of $p(\theta_1, \theta_2 \mid y)$ for the candidate priors, namely, the horseshoe+ (HSPlus), the horseshoe (HS), Laplace, normal, pure-local, pure-global and the two reference priors for the product mean and ratio of two means (Fieller-Creasy) problems. }
\label{fig:fc_plot_theta_density}%
\end{figure}
%Figure~\ref{fig:fc_plot_theta_density-b} shows the histograms of the posterior draws within a 85\% interval for the product $\theta_1\theta_2$ for the prior and posterior distributions, and Figure~\ref{fig:fc_plot_theta_density-c} draws the same diagram for the ratio $\theta_1/\theta_2$ for the prior and posterior distributions.
\begin{table}[ht!]
\centering
\def~{\hphantom{0}}
\caption{Posterior quantiles for the marginal parameters of interest $\psi_1 = \theta_1/\theta_2$ and $\psi_2 = \theta_1 \theta_2$ for the candidate priors, namely, the horseshoe+ (HS+), the horseshoe (HS), Laplace, normal, pure-local, pure-global and the reference priors. All entries in the table are multiplied by $1,000$.}{%
{\footnotesize
\begin{tabular}{rrrrrrrr}
& \multicolumn{3}{c}{$\psi_1 = \theta_1/\theta_2$} & & \multicolumn{3}{c}{$\psi_2 = \theta_1 \theta_2 $} \\
& $Q_1$ & Median & $Q_3$ & & $Q_1$ & Median & $Q_3$ \\
HS+ & -1020 & -24${\cdot}$8 & 696${\cdot}$7 & HS+ & -0${\cdot}$6 & -0${\cdot}$0 & 0${\cdot}$2 \\
HS & -941${\cdot}$3 & -37${\cdot}$8 & 689${\cdot}$7 & HS & -1${\cdot}$1 & -0${\cdot}$0 & 0${\cdot}$5 \\
Laplace & -1092 & -170${\cdot}$2 & 618${\cdot}$1 & Laplace & -5${\cdot}$2 & -0${\cdot}$4 & 2${\cdot}$1 \\
Normal & -1060 & -250${\cdot}$3 & 482${\cdot}$2 & Normal & -8${\cdot}$6 & -1${\cdot}$3 & 2${\cdot}$4 \\
Local & -999${\cdot}$6 & -126${\cdot}$7 & 489${\cdot}$2 & Local & -3${\cdot}$9 & -0${\cdot}$2 & 1${\cdot}$3 \\
Global & -1092 & -120${\cdot}$5 & 812${\cdot}$1 & Global & -2${\cdot}$1 & -0${\cdot}$0 & 0${\cdot}$8 \\
Reference & -71${\cdot}$9 & -8${\cdot}$5 & 58${\cdot}$7 & Reference & -14${\cdot}$6 & -3${\cdot}$8 & 3${\cdot}$9 \\
\end{tabular}%
}%
}
\label{tab:fc-and-pm}
%\begin{tabnote}
%HS, Horseshoe.
%\end{tabnote}
\end{table}
%%%%%%%%%%%%% THEORY %%%%%%
\section{Posterior contraction for the sum of squares problem}\label{sec:post-conc}
Here we provide some theoretical basis to the behavior of horseshoe prior for the sum of squares problem. It turns out that a sufficient statistic for $\psi = \sum_{i=1}^{p} \theta_i^2$ is $Z = \sum_{i=1}^{p} y_i^2$ and $Z$ follows a non-central $\chi^2$ distribution with $p$ degrees of freedom and non-centrality parameter $\psi$ with density:
\begin{equation*}
f_Z(z \mid \psi) = \frac{1}{2} e^{-(z+\psi)/2}\left(\frac{z}{\psi} \right)^{p/4-1/2} I_{p/2-1}(\sqrt{\psi z}),
\end{equation*}
where $I_{\nu}(z)$ is the modified Bessel function of the first kind. This allows us to investigate the properties of the posterior under two different priors on the non-centrality parameter $\psi$, namely, the prior induced on $\psi$ by putting a horseshoe prior and a normal prior on each parameter $\theta_i$. The limiting form $I_{\nu}(z) \sim e^z/\sqrt{2\pi z}$ when $\nu$ is fixed and $z \to \infty$ \citep[Chapter 10.30 (ii) of][]{Olver:2010:NHMF} implies
\begin{equation}
f_Z(z \mid \psi) \approx \frac{1}{2\sqrt{2 \pi}} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}z^{(p-3)/4}\psi^{-(p-1)/4},\label{eq:lik}
\end{equation}
for large $z$. We can use this approximation to prove the following:
\begin{theorem}\label{th:post-ss}
The posteriors for $\psi$ as $Z\to \infty$ under the horseshoe prior and the gamma prior induced by normal prior on $\theta_i$s are approximately given by
\begin{align}
p_{HS}(\psi \mid Z) & \propto e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{-(p+1)/4} \frac{1}{(\psi+p)}, \label{eq:hs_ss_post} \\
p_{N}(\psi \mid Z) & \propto e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{(p-3)/4} e^{ - \frac{\psi}{2 \tau^2} }. \label{eq:psi-norm}
\end{align}
\end{theorem}
\begin{figure}[h!]
\centering
\includegraphics[width=0.7\linewidth,height=2in]{revised-psi-hist-density-bw.eps}
\caption{Kernel density estimates (shaded regions) of the posterior $p(\psi \mid Z)$ under the half-Cauchy and the $\Nor(0,300)$ prior on each $\theta_i$ along with their Laplace approximations (lines) given by \eqref{eq:hs_ss_post} and \eqref{eq:psi-norm}. The darker shade and dotted line denote the $\Nor(0,300)$ prior and the lighter shade and solid line denote the half-Cauchy prior. }
\label{fig:comp.psi}
\end{figure}
%A proof is given in the Supplementary \S S.2.1.%Appendix~\ref{app:th:post-ss}
The main difference between the horseshoe prior and the gamma prior is that the posterior density under the horseshoe prior is non-zero at $\psi=0$, unlike the posterior density under the gamma prior. In fact, the posterior density under the horseshoe prior has a pole at $\psi=0$ that shrinks the posterior mean $E(\psi \mid Z)$ towards the origin. The kernel density estimates of both the posterior densities are shown in Fig. \ref{fig:comp.psi}, along with their approximations derived in \eqref{eq:hs_ss_post} and \eqref{eq:psi-norm}. The plot serves two purposes. First, it shows the accuracy of the Laplace approximation. Secondly, it shows that the posterior density under horseshoe prior and the gamma prior concentrates near $\psi = 100$ and $\psi = 300$, and the two posterior distributions concentrates on two disjoint regions, namely, horseshoe on $(0,Z)$ and gamma on $(Z,\infty)$. In fact, this can be proved theoretically. We shall now show that the posterior distribution of $\psi$ for horseshoe prior is going to put almost all of its density in the interval $(0,Z)$, and the posterior for gamma prior is contained almost entirely in the region $[Z, \infty)$ for large values of $\tau^2$.
\begin{theorem}\label{th:horse-ss}
Let $[y_i \mid \theta_i] \sim \Nor(\theta_i,1)$ and $\psi = \sum_{i=1}^{p} \theta_i^2$, and $Z = \sum_{i=1}^{p} y_i^2$. Let the prior on $\theta_i$ be the horseshoe prior given by the hierarchy: $\theta_i \sim \Nor(0,\tau^2)$ and $\tau \sim C^+(0,1)$. Then, for large $Z$ the posterior distribution of $\psi$ given $Z$ is concentrated on the interval $(0,Z)$, that is,
$$
P(\psi \geq Z \mid Z ) \leq 4(p+1)^{-1} \mbox{ as } Z \to \infty.
$$
\end{theorem}
%Appendix~\ref{app:th:horse-ss}
Now we show that the posterior mean under the gamma prior induced by a component-wise $\Nor(0,\tau^2)$ converges to $\sum_{i=1}^{p} y_i^2 + p$ as $\tau^2 \to \infty$. In fact, we prove a stronger result that $p(\psi \mid Z)$ shrinks away from the interval $(0,Z)$ as $p \to \infty$ if $\tau^2$ is large enough such that $\sum_{i=1}^{p} y_i^2 < 2\tau^2$.
\begin{theorem}\label{th:norm-ss}
Let $[y_i \mid \theta_i] \sim \Nor(\theta_i,1)$ and $\psi = \sum_{i=1}^{p} \theta_i^2$, and $Z = \sum_{i=1}^{p} y_i^2$. Let the prior on $\theta_i$ be the normal prior: $\theta_i \sim \Nor(0,\tau^2)$ where $\tau^2$ is large such that $Z \leq 2\tau^2$. Then, for large $p$ the posterior distribution of $\psi$ given $Z$ is concentrated on the interval $(Z,\infty)$, that is,
$$
P(\psi \leq Z \mid Z ) \leq \frac{1}{\Gamma(\frac{p+5}{4})} \left(\frac{Z}{2\tau^2} \right)^{\frac{p+1}{4}} \mbox{ as } p \to \infty.
$$
\end{theorem}
%A proof is given in Supplementary \S S.2.3. %Appendix~\ref{app:th:norm-ss}.
The proofs of the theorems are given in the Supplementary Material. We conjecture similar concentration results will hold for the horseshoe+ prior, although an investigation is beyond the scope of the current article.
%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}\label{sec:discuss}
Global-local shrinkage priors such as the horseshoe and horseshoe+ priors were initially designed as a computationally attractive alternative to spike-and-slab priors for the sparse signal recovery problem, where pure-global shrinkage estimators such as the James-Stein estimator perform poorly \citep{polson2010shrink}. Several optimality results are available for the estimation of normal means using these priors \cite[for example,][]{van2014horseshoe,ghosh2015asymptotic}. The goal of the current article is to demonstrate that these priors go beyond estimating just the normal means; they also perform well in estimating low-dimensional, nonlinear functions of interest. We show numerically that the class of regularly varying priors performs as well as Bernardo's reference prior in the bivariate product and ratio problems, and better for the sum of squares problem in a sparse setting, and thus satisfies the informal criterion for a candidate default prior ``which never leads to an obviously stupid answer'' \citep{scott2014comment}. On one hand, the reference priors are quite general in that they formally maximize the divergence between the prior and the posterior, and it is not clear whether the regular varying global-local priors possess any such properties. On the other hand, (a) the computing reference prior can be very cumbersome for many different problems (and may not even be possible), (b) if one changes the parameterization reference prior needs to be recomputed, (c) reference priors are often not integrable, making them unsuitable for model selection purposes and (d) reference priors depend on an ordering of the model parameters, even in cases where no such natural ordering exists. A formal non-informative prior is not always appropriate or useful, for example when the true $\theta$ is sparse \citep{scott2006exploration}, where the global-local shrinkage prior is optimal in terms of Bayes risk and asymptotic minimaxity \citep{van2014horseshoe,ghosh2015asymptotic}. The global-local priors also come with the added benefit of always being proper. Thus, it can be intuitively argued that the regularly varying global-local priors offer a practical compromise between Bernardo's formal framework and flat priors. It should be pointed out that although regular variation, along with global-local shrinkage, appears to be a key property for success in multi-parameter problems, a thorough formal investigation should be considered future work.
%{\color{red} (This paragraph puts far too much emphasis on an unpublished manuscript, that Biometrika does not allow us to refer. Consider deleting references to Sims, perhaps the entire paragraph can be deleted)} A similar phenomenon happens in the case of Bayesian analysis of instrumental variables as illustrated in a recent technical report by C. Sims, who observed that in cases of weak instruments or large $p$, small $n$ regression, a prior with Gaussian or higher order polynomial tails dominates the likelihood with fat tails, even if the likelihood has a sharp mode not very far from the prior mode. This is illustrated in Figure 1 of Sims' technical report with a Cauchy likelihood and a t-prior with $9$ degrees of freedom. Sims shows that when the prior location shifts while keeping the likelihood fixed, at first the posterior moves away from the prior but comes back and coincides with the prior mode when the departure becomes large. This phenomenon is explained by the Theorem 3.3 and Corollary 3.4 of \citet{andrade2011bayesian}, where they show that assigning a prior which is lighter tailed than the likelihood $f$, we obtain a posterior distribution based on the prior distribution, such that the observation $y$ is wholly rejected. Conversely, if the prior is heavier tailed than $f$, we obtain a posterior distribution based on the data distribution, so that now it is the prior information that is rejected. Like the $\Nor(0,300)$ prior that leads to unreasonable estimates for \cite{efron1973discussion}, Sims shows that even a weak $\Nor(0,100 I)$ prior has a dramatic effect on the posterior. Indeed, the improper priors Sims uses towards this conflict resolution has a heavier tail than the likelihood, letting the data distribution determine the posterior tails. We conjecture that for these important applied problems outlined by Sims, regularly varying priors should lead to a more sensible answer for weak instruments or multi-parameter problems, compared to the proper flat priors.
\section*{Acknowledgment}
The authors thank the Editor, the Associate Editor and two anonymous referees for their constructive suggestions. The second author was supported by the U.S. National Science Foundation through the Statistical and Applied Mathematical Sciences Institute.
%\pagenumbering{Alph}
\beginsupplement{
\section*{Supplementary material}
\label{SM}
The supplementary file contains more details on regular variation, the proofs of theorems in \S \ref{sec:post-conc}, sampling strategies for the global-local shrinkage priors, additional simulation results and a discussion of connections with the penalized complexity priors.
\section{Regular variation}\label{sec:regvar}
The concept of regular variation is one of the most used tools in applied probability and is discussed in great details in Chapter VIII of \citet{feller1971} and by \citet{bingham1989regular}.
\begin{Def}
A measurable function $f(\cdot)$ is regularly varying at $\infty$ if with index $\alpha \in \Re$, denoted as $f \in R_{\alpha}$, if
\beq
\lim_{x \to \infty} \frac{f(\lambda x)}{f(x)} \to \lambda^\alpha, \quad \forall \lambda > 0. \label{eq:regvar}
\eeq
Similarly, a measurable function $f(\cdot): \Re^{+}\to \Re^{+} $ is regularly varying at the origin if with index $\alpha \in \Re$, denoted as $f \in R_{\alpha}(0+)$, if
\beq
\lim_{x \to 0+} \frac{f(\lambda x)}{f(x)} \to \lambda^\alpha, \quad \forall \lambda > 0. \label{eq:regvar0}
\eeq
\end{Def}
Using the characterization theorem in \citet{bingham1989regular}, it can be shown that, $f \in R_{\alpha}$ if and only if $f(x) = x^{\alpha}l(x)$, where $l \in R_0$, and such a function $l(\cdot)$ is called a slowly-varying function. One can analogously define rapid variation where a measurable function $f$ is rapidly varying with index $\infty$ when the limit of the quantity on the left hand side of Equation (\ref{eq:regvar}) goes to infinity.
\begin{Def}
A random variable $Z$ has a regularly varying right tail if with index $\alpha \in \Re$ if its distribution (say $F_Z$) satisfies:
\begin{equation*}
1-F_Z(x) = P(Z > x) = x^{-\alpha} l(x), \quad \mbox{for } x > 0.
\end{equation*}
In other words, $Z \sim F_Z$ is $RV(\alpha)$ if $\bar{F}_Z \in R_{-\alpha}$.
\end{Def}
One of the most important properties of regular variation is that it is preserved under many operators that commonly arise in inferential problems where we are interested in a many-to-one transformation of a parameter vector. A few of the closure properties that appear in Proposition 1.5.7 of \citet{bingham1989regular} are listed below.
\begin{proposition}\label{lemma:closure}
\begin{enumerate}
\item if $f(x) \in R_{\alpha}$, then $f(x)^\rho \in R_{\rho \alpha}$.
\item if $f_i \in R_{\alpha_i}$, for $i = 1,2$, then $f_1+ f_2 \in R_{\alpha}$ where $\alpha = \max\{\alpha_1, \alpha_2\}$.
\item $f_i \in R_{\alpha_i}$, for $i = 1,\ldots,k$, and if $r(x_1, \ldots, x_k)$ is a rational function with positive coefficients then $r(f_1(x), \ldots, f_k(x)) \in R_{\alpha}$ for some $\alpha$.
\end{enumerate}
\end{proposition}
Similar closure properties can be proved for functions of random variable with regular varying right tails. For example, Proposition \ref{lemma:closure} B. implies the following: if $Z_1$,$Z_2$ are independent and have distributions with regularly varying right tails with coefficients $\alpha_1$ and $\alpha_2$, then $Z_1+Z_2$ will have a regularly varying right tail with coefficient $\min(\alpha_1,\alpha_2)$. The closure properties hold true for many different many-to-one functions such as the sum, maximum, product and ratio of random variables. The four classical problems described in \S 3 of our paper can be then recast as problems of regular variation on the prior $p(\psi)$, where the marginal parameters of interest $\psi = \psi(\theta)$ satisfies the closure properties.
%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Proofs of theorems}
\subsection{Proof of Theorem \ref{th:post-ss}}
\begin{proof}
Under the horseshoe prior, the prior on $\psi$ is proportional to
$$
p( \psi ) \propto \int_{0}^{\infty} \psi^{p/2-1} \exp(-\psi/(2\tau^2)) \tau^{-p} (\tau^2+1)^{-1} d\tau.
$$
We can approximate the above integral using Laplace approximation as
\begin{align*}
p(\psi) & \propto \psi^{p/2-1} \int_0^{\infty} e^{-p\left(\frac{\psi}{2\tau^2 p}+\log \tau \right)} (\tau^2+1)^{-1} d\tau \\
& = \psi^{p/2-1} \int_0^{\infty} e^{M g(\tau)} h(\tau) d\tau,
\end{align*}
where $M = p$, $g(\tau) = -\left(\psi (2\tau^2 p)^{-1}+\log \tau \right)$ and $h(\tau) = (\tau^2+1)^{-1} > 0$. The unique maximum of $g(\cdot)$ in $[0,\infty)$ is $\tau_0 = \sqrt{\psi/p}$, and $|g\rq{}\rq{}(\tau_0)|= 2p/\psi$. Hence, the Laplace approximation for the integral can be written as up to a proportionality constant independent of $\psi$ as
\begin{align*}
p(\psi) & \propto \psi^{p/2-1} \sqrt{\frac{2\pi}{M|g\rq{}\rq{}(\tau_0)|}} h(\tau_0) e^{M g(\tau_0)} \\
& \propto \psi^{p/2-1} \sqrt{\frac{2\pi}{|p/\psi|}} h(\tau_0) e^{-p(1/2+1/2 \log(\psi/p))} \\
& \propto \psi^{-\half} h(\psi/p) \propto \frac{1}{\sqrt{\psi} (\psi+p)} .
\end{align*}
On the other hand, under the normal prior on $\theta$ the prior on $\psi$ is proportional to $p(\psi) \propto \psi^{ \frac{p}{2} -1 } e^{ - \frac{\psi}{2 \tau^2} }$. Hence, noting the likelihood from Equation (4), the posteriors on $\psi$ when $Z\to \infty$ under the horseshoe prior and the gamma (induced by normal prior on $\theta_i$s) are given as:
\begin{align*}
p_{HS}(\psi | Z) & \propto e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{-(p+1)/4} \frac{1}{(\psi+p)} ,\\
p_{N}(\psi | Z) & \propto e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{(p-3)/4} e^{ - \frac{\psi}{2 \tau^2} }.
\end{align*}
\end{proof}
%\begin{remark} \textbf{A Regular Variation Argument}:
%We can also arrive at \eqref{eq:hs_ss_post} using a regular variation (hereafter abbreviated as RV) approach.
%The key insight here is that if the prior $ p(\eta)$ where $ \eta =1+\lambda$ has RV, then so does the marginal $m(y) $ \cite[Theorem 5.1 of ][]{barndorff1982normal} and the posterior $p(\eta|y)$ is driven by the likelihood \cite{dawid1973posterior}. Let us focus on the horseshoe posterior for an univariate parameter $\theta$ for $p=1$.
%\begin{eqnarray*}
%p(\theta |y) &=& \frac{e^{-\frac{1}{2}(\theta - y)^2}}{m(y)} p(\theta).
%\end{eqnarray*}
%Writing the prior on $\theta$ as a scale mixture of normal we have
%\begin{eqnarray*}
%p(\theta |y) &=& \frac{e^{-\frac{1}{2}(\theta - y)^2}}{m(y)} \int_{0}^{\infty} \exp\left(-\frac{\theta^2}{2 \lambda^2}\right) \frac{1}{\sqrt{2\pi \lambda^2}} \frac{1}{1+\lambda^2} d\lambda.
%\end{eqnarray*}
%Let $\lambda^2 = u$. Then
%\begin{eqnarray*}
%p(\theta |y) &=& \frac{e^{-\frac{1}{2}(\theta - y)^2}}{m(y)} \int_{0}^{\infty} \exp\left(-\frac{\theta^2}{2 u}\right) \frac{1}{\sqrt{2\pi u}} \frac{1}{1+u} u^{-1/2}du\\
%&=& \frac{e^{-\frac{1}{2}(\theta - y)^2}}{m(y)} \int_{0}^{\infty} \exp\left(-\frac{\theta^2}{2 u}\right) \frac{1}{\sqrt{2\pi u}} \frac{u}{1+u} u^{-3/2}du.
%\end{eqnarray*}
%Since $L(u)=u/(1+u)$ is slowly varying, we have $f(u)=u^{-3/2} u/(1+u)$ as in \citet{barndorff1982normal} Theorem 5.2. Thus, using Equation (6.2) of \citet{barndorff1982normal}, we have that as $|\theta| \to \infty$
%\begin{eqnarray*}
%p(\theta |y) &=& \frac{e^{-\frac{1}{2}(\theta - y)^2}}{m(y)} \theta^{-2} \frac{\theta^2}{1 + \theta^2} = \frac{e^{-\frac{1}{2}(\theta - y)^2}}{m(y)} \frac{1}{1 + \theta^2}.
%\end{eqnarray*}
%Let $\theta^2 = \psi$ and $y^2 = z$. Then,
%\begin{eqnarray*}
%p(\psi | z) &=& \frac{e^{-\frac{1}{2}(\sqrt{\psi} - \sqrt{z})^2}}{m(z)} \frac{\psi^{-1/2}}{1+\psi},
%\end{eqnarray*}
%which is exactly the same for $p=1$ case for \eqref{eq:hs_ss_post}. For $p>1$, the posteriors $p(\theta_i | y_i)$ are independent, but not identically distributed.
%\end{remark}
\subsection{Proof of Theorem \ref{th:horse-ss}}
\begin{proof}
\begin{align*}
P_{HS}(\psi \geq Z | Z) &\leq \frac{\int_{Z}^{\infty} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi }{\int_{0}^{\infty} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi} \\
& \leq \frac{\int_{Z}^{\infty} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi }{\int_{0}^{Z} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi}.
\end{align*}
Now, note that $\exp\{{-(\sqrt{z}-\sqrt{\psi})^2/2}\}$ is a decreasing function, which means it can be bounded by its values at the boundaries. Hence,
\begin{align}
P_{HS}(\psi \geq Z | Z) & \leq \frac{\int_{Z}^{\infty} \psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi }{\int_{0}^{Z} \psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi} .\label{eq:tailprob}
\end{align}
Now, the integral in the denominator can be trivially bounded as:
\beq
\int_{0}^{Z} \psi^{-(p+1)/4} \frac{1}{(\psi+p)} d\psi \geq Z^{-(p-3)/4} \frac{1}{Z+p} .\label{eq:lower}
\eeq
For upper bounding the numerator integral, we invoke Karamata's integral theorem, stated below. A proof can be found in \citet{bingham1989regular}.
\begin{theorem} (Karamata's theorem).
Let $L$ be a slowly varying and locally bounded function in $[x_0, \infty]$ for some $x_0 \geq 0$, then
\ben
\item for $\alpha > -1$,
$$
\int_{x_0}^{x} t^{\alpha}L(t) dt \sim (\alpha+1)^{-1} x^{\alpha+1} L(x), \quad x \rightarrow \infty ,
$$
\item for $\alpha < -1 $,
$$
\int_{x}^{\infty} t^{\alpha}L(t) dt \sim -(\alpha+1)^{-1} x^{\alpha+1} L(x), \quad x \rightarrow \infty .
$$
\een
\end{theorem}
Using the second part of Karamata\rq{}s integral theorem and the fact that $L(\psi) = \psi (\psi +p)^{-1}$ is a slowly varying function, we can derive an approximation for the numerator integral:
\beq
\int_{Z}^{\infty} \psi^{-(p+1)/4-1} \frac{\psi}{(\psi+p)} d\psi \sim \frac{4}{p+1} Z^{-(p-3)/4} \frac{1}{Z+p} \quad \mbox{ as } Z \rightarrow \infty. \label{eq:upper}
\eeq
Putting \eqref{eq:lower} and \eqref{eq:upper} together, we can derive an upper bound for the tail probability in \eqref{eq:tailprob} as follows:
\begin{align*}
P_{HS}(\psi \geq Z | Z) & \leq \frac{4}{p+1} .
\end{align*}
\end{proof}
%So, we have proved the following theorem:
\subsection{Proof of Theorem \ref{th:norm-ss}}
\begin{proof}
To prove this assertion, we try to find a diminishing upper bound to the probability $P_{N}(\psi \leq Z | Z)$ using known inequalities. First, note as in Theorem~5.2, $e^{-(\sqrt{z}-\sqrt{\psi})^2/2}$ is a decreasing function and can be upper and lower bounded by its boundary values.
\begin{align}
P_{N}(\psi \leq Z | Z) & \leq \frac{\int_{0}^{Z} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{(p-3)/4} e^{-\psi/2\tau^2} d\psi }{\int_{0}^{\infty} e^{-(\sqrt{z}-\sqrt{\psi})^2/2}\psi^{(p-3)/4} e^{-\psi/2\tau^2} d\psi} \nonumber\\
& \leq \frac{\int_{0}^{Z} \psi^{(p-3)/4} e^{-\psi/2\tau^2} d\psi }{\int_{0}^{\infty} \psi^{(p-3)/4} e^{-\psi/2\tau^2} d\psi} = \frac{1}{\Gamma(\frac{p+1}{4})} \gamma \left(\frac{p+1}{4}, \frac{Z}{2\tau^2} \right)\nonumber \\
& \leq \frac{1}{\Gamma(\frac{p+1}{4})} \frac{ \left(\frac{Z}{2\tau^2} \right)^{\frac{p+1}{4}} }{\frac{p+1}{4} }
= \frac{1}{\Gamma(\frac{p+5}{4})} \left(\frac{Z}{2\tau^2} \right)^{\frac{p+1}{4}}, \label{eq:upper2}
\end{align}
where the last inequality follows from the well-known upper bound for the lower incomplete gamma function:
\beq
\gamma(a,x) \leq \frac{x^{a-1}}{a}(1-e^{-x}) \leq \frac{x^{a}}{a} \quad \mbox{ for } x>0, a> 1 \nonumber.
\eeq
Clearly, the upper bound in Equation \eqref{eq:upper2} goes to zero if $Z \tau^{-2} \rightarrow 0$. In fact, the upper bound goes to zero if $Z \leq 2\tau^2$ and $p \rightarrow \infty$.
%Now, we are at stage to state the posterior concentration theorem for the usual Gamma prior as follows:
\end{proof}
%%%%%%%%%%%%%%% Sampling Strategy Moved to Appendix
\section{Sampling Schemes for global-local priors}\label{sec:sampling}
Here we discuss two possible sampling strategies for the full Bayes inference under the horseshoe and the horseshoe+ priors.
\subsection{Exponential slice sampling for the horseshoe posterior}\label{sec:slice}
The horseshoe model is a hierarchical sparsity model. We need to infer the location parameter $(\theta_1, \ldots , \theta_p)$
given data $y=(y_1, \ldots , y_p)$. Specifically, suppose that
\begin{eqnarray*}
(y_i \mid \theta_i ) & \sim & \Norm (\theta_i,1),\; (\theta_i \mid \lambda_i^2) \sim \Norm (0, \lambda_i^2 \tau^2), \\
\lambda_i &\sim& C^+(0,1),\; \tau \sim C^+(0,1),
\end{eqnarray*}
where $C^+(0,1)$ is a standard half-Cauchy distribution, so that:
$$
p( \lambda_i ) = \frac{2}{\pi} \frac{1}{ 1 + \lambda_i^2 } \; \; {\rm and} \; \; p(\tau) = \frac{2}{\pi} \frac{1}{1+\tau^2}.
$$
This can be re-parameterised in terms of shrinkage factors where $ \kappa_i = (1 + \lambda_i^2 \tau^2 )^{-1} $. We can marginalise out $\theta_i$ and write the joint posterior as:
$$
p( \kappa_1 , \ldots , \kappa_p , \tau^2 | y ) \propto
\frac{\tau^p}{1+\tau^2} \prod_{i=1}^p \frac{ e^{- \kappa_i \frac{y_i^2}{2}}}{\sqrt{1-\kappa_i}} \frac{1}{\kappa_i(\tau^2-1) + 1}.
$$
Now introduce latent variables $ (\omega_1 , \ldots , \omega_p)$ and $\omega$ to write
\begin{align*}
\frac{1}{\kappa_i(\tau^2-1) + 1} & = \int_0^\infty e^{- \omega_i (\kappa_i(\tau^2-1) + 1)} d \omega_i, \\
\frac{1}{\tau^2+1} & =\int_0^\infty e^{- \omega (\tau^2+1)} d \omega.
\end{align*}
Therefore, we have an augmented joint posterior:
$$
p( \kappa_1 , \ldots , \kappa_p , \omega_1 , \ldots , \omega_p, \tau^2 , \omega | y ) \propto
\tau^p e^{- \omega (\tau^2+1)} e^{- \sum_{i=1}^p \kappa_i ( \omega_i (\tau^2-1) + y_i^2/2) - \omega_i} \prod_{i=1}^p \frac{1}{\sqrt{1-\kappa_i}} .
$$
Now introduce slice variables $(u_1, \ldots , u_p)$ on the set $\mathbb{I} \{0 < u_i < ( 1 -\kappa_i)^{-\frac{1}{2}} \}$. Therefore we have a joint posterior proportional to:
$$
\tau^p e^{- \omega (\tau^2+1)} e^{- \sum_{i=1}^p \kappa_i ( \omega_i (\tau^2-1) + y_i^2/2) - \omega_i} \prod_{i=1}^p
\mathbb{I}\{0 < u_i < ( 1 -\kappa_i)^{-\frac{1}{2}} \}.
$$
All the Gibbs conditionals are known in closed form.
$$
( \kappa_i | \omega_i , u_i, \tau^2) \sim \mbox{Exp} \left ( \omega_i (\tau^2-1) + y_i^2/2 \right ) \mathbb{I} \{ 1 > \kappa_i > \max(1 - u_i^{-2},0) \},
$$
is a truncated exponential distribution.
$ ( \omega_i | \kappa_i , \tau^2 ) \sim \mbox{Exp} ( \kappa_i(\tau^2-1) + 1 ) $ and $ ( \omega | \tau^2 ) \sim \mbox{Exp} ( \tau^2 + 1 ) $ are exponentially distributed.
$ ( u_i| \kappa_i ) \sim \mbox{U} \{0 , ( 1 -\kappa_i)^{-\frac{1}{2}}\} $ follows a uniform distribution.
$ ( \tau^2 | \kappa , \omega ) \sim Ga \{ (p+1)/2 , \omega + \sum_{i=1}^p \kappa_i \omega_i \}$ follows a Gamma distribution.
We can then simulate for the posterior $(\theta_i | y) $ using
$$
( \theta_i | \kappa_i , y_i ) \sim \Norm \left( ( 1 - \kappa_i)y_i , 1-\kappa_i \right).
$$
%In all four cases, horseshoe provides a ``reasonable'' answer that matched Efron's intuition. The std. error $ sd ( \hat{\psi} ) $ is heavier tailed.
\subsection{Truncated normal slice sampling for the horseshoe+ posterior}\label{sec:trunc}
\citet{bhadra2015horseshoe+} show that for horseshoe+, the marginal prior on $\lambda_i$ is given by
$$
p( \lambda_i | \tau ) = \left ( \frac{4}{\pi^2 \tau} \right ) \frac{ \ln \left \{ ( \lambda_i / \tau )^2 \right \} }{ ( \lambda_i / \tau )^2 -1}
\; \; {\rm where} \; \; \tau \sim p(\tau) \; .
$$
The posterior is proportional to
$$
p( \lambda_1 , \ldots , \lambda_p , \tau | y ) \propto \left \{ \prod_{i=1}^{p} \frac{1}{\sqrt{1+\lambda_i^2}} e^{-\frac{y_i^2}{2} \frac{1}{1+\lambda_i^2}}
\frac{\log(\lambda_i/\tau)}{(\lambda_i/\tau)^2-1} \right \} \tau^{-p} p( \tau ).
$$
To simulate, we use the slice sampling and the Gibbs sampling. We introduce a vector of slice variables $ u =(u_1, \ldots , u_p)$ and simulate the joint distribution
$$
p( \lambda_1, \ldots , \lambda_p , u_1 , \ldots , u_p , \tau | y ) \propto
\left \{ \prod_{i=1}^{p} \frac{1}{\sqrt{1+\lambda_i^2}} e^{-\frac{y_i^2}{2} \frac{1}{1+\lambda_i^2}}
\mathbb{I}_{ \left\{ f(u_i) > \lambda_i/\tau \right\}} \right \} \tau^{-p} p(\tau) ,
$$
where $f^{-1}(x) = \log x / (x^2-1)$ is a monotonically decreasing function and so has a well defined inverse. If we marginalize out $ u$ then we have the required marginal. The complete conditional distributions are
\begin{align*}
p(\lambda_i | u_i,\tau, y) &\propto (1+\lambda_i^2)^{-1/2} \mbox{exp}[ -y_i^2/\{2(1+\lambda_i^2)\}] \mathbb{I}_{ \left\{ \lambda_i < \tau f(u_i) \right\}},\\
p(u_i | \lambda_i,\tau, y) &\propto \mathbb{I}_{\left\{ 0 < u_i < f^{-1}(\lambda_i/\tau) \right\}},\\
p(\tau | \lambda_i,u_i, y) &\propto \tau^{-p} p(\tau) \mathbb{I}_{ \left\{ \tau > \lambda_i / f(u_i) \; \forall i \right\}}.
\end{align*}
The last conditional is $ p(\tau | \tau > \max ( \lambda_i / f(u_i) ) ) $. Posterior draws of $\theta$ can then be obtained by setting $\kappa_i = (1+ \lambda_i^2 \tau^2)^{-1}$ and
$$
( \theta_i | \kappa_i , y_i ) \sim \Norm \left ( ( 1 - \kappa_i)y_i , 1-\kappa_i \right ).
$$
\section{MCMC Convergence Diagnostics}
\begin{figure}[ht!]%
\centering
\includegraphics[width = \linewidth,height=3in]{diag_bivar_0426.eps}%
\caption{The convergence diagnostics plot for the bivariate examples under the horseshoe+ prior. The four panels, clockwise from top left, show the trace-plot, the density plot, the running mean plot and the auto-correlation plot.}%
\label{fig:diag}%
\end{figure}
We provide a few diagnostic plots that ensures the convergence of the Markov chain Monte Carlo sampling under the global-local shrinkage priors. For ease of illustration, we pick the bivariate case where the true values of the parameters $(\theta_1, \theta_2)$ is set at the origin $(0,0)$ and the horseshoe+ prior, as its convergence has not been studied elsewhere. We show four plots in Fig \ref{fig:diag} essential to judge convergence: the trace plot, the density plot, the running mean plot and the auto-correlation plot. The trace-plot shows the posterior samples as a time-series and the white-noise like plot in Fig \ref{fig:diag} ensures within-chain convergence or good mixing. The running mean plot shows the running mean of the sampling process shown in trace-plot and our plot suggests fast convergence to the target distribution. The density plot also shows agreement between the target distribution by different chains and finally the auto-corrleation plot in Fig \ref{fig:diag} shows low correlation between successive draws and good mixing.
\section{Performance for Non-Gaussian Observations}
As suggested by a referee, we have investigated bivariate examples to see how well the the results extend beyond the Gaussian observation model. To this end, we simulated $K=100$ bivariate observations as independent draws from a $t$-distribution with $3$ degrees of freedom on each component of $(\theta_1, \theta_2)$.
\begin{align*}
(Y_{k,i} \mid \theta_i) &\sim t (d.f. = 3, n.c.p = \theta_i), i = 1,2, k = 1, \ldots, K.
\end{align*}
We restrict ourselves to the case where the true values for both $\theta_1$ and $\theta_2$ are zero, that is, the observations are drawn from a central $t$-distribution with 3 degrees of freedom on each component. The sample mean for the data is $(\bar{y}_1, \bar{y}_2) = (0{\cdot}043,-0{\cdot}084)$. The results are given in Fig. \ref{fig:t_fc_plot_theta_density} and Table \ref{tab:t-fc-and-pm}. It turns out that even in this case the global-local priors achieve stronger concentration near the origin compared to many of the other alternatives.
\begin{figure}[ht!]%
\centering
\includegraphics[width = 0.8\linewidth,height = 2.5in]{t-8-contour-plots-grayscale-legend.eps}%
\caption{Two-dimensional contour plots of $p(\theta_1, \theta_2 \mid y)$ for the six shrinkage priors, namely, the horseshoe+ (HSPlus), the horseshoe (HS), Laplace, normal, pure-local, pure-global and the two reference priors for product mean and ratio of two means (Fieller-Creasy). }%
\label{fig:t_fc_plot_theta_density}%
\end{figure}
\begin{table}
\centering
\def~{\hphantom{0}}
\caption{Posterior quantiles for the marginal parameters of interest $\psi_1 = \theta_1/\theta_2$ (the Fieller--Creasy problem) and $\psi_2 = \theta_1 \theta_2$ for the candidate priors. All entries in the table have been multiplied by $1,000$ for the sake of representation.} {%
{\footnotesize
\begin{tabular}{rrrrrrrr}
& \multicolumn{3}{c}{$\psi = \theta_1/\theta_2$} & & \multicolumn{3}{c}{$\psi = \theta_1 \theta_2 $} \\
& $Q_1$ & Median & $Q_3$ & & $Q_1$ & Median & $Q_3$ \\
HS+ & -5010 & -628${\cdot}$8 & 3776 & HS+ & -4${\cdot}$7 & -0${\cdot}$2 & 1${\cdot}$2 \\
HS & -3450 & -739${\cdot}$3 & 2273 & HS & -7${\cdot}$1 & -0${\cdot}$5 & 1${\cdot}$5 \\
Laplace & -3057 & -1316 & 1270 & Laplace & -17${\cdot}$9 & -5${\cdot}$5 & 1${\cdot}$4 \\
Normal & -3061 & -1219 & 1928 & Normal & -24${\cdot}$8 & -6${\cdot}$7 & 5${\cdot}$1 \\
Local & -3541 & -1007 & 2662 & Local & -11${\cdot}$9 & -1${\cdot}$6 & 3${\cdot}$0 \\
Global & -2669 & -875${\cdot}$8 & 1816 & Global & -13${\cdot}$3 & -1${\cdot}$7 & 3${\cdot}$1 \\
Reference & -89${\cdot}$5 & -18${\cdot}$6 & 53${\cdot}$8 & Reference & -30${\cdot}$4 & -10${\cdot}$0 & 5${\cdot}$4 \\
\end{tabular}%
}%
}
\label{tab:t-fc-and-pm}
%\begin{tabnote}
%HS, Horseshoe.
%\end{tabnote}
\end{table}
\section{Hyper-parameter for for $\tau$}
We performed a prior sensitivity analysis to justify the choice of hyper-parameter $\eta=1$ for the scale of the half-Cauchy prior on $\tau$ in the hierarchical model for the horseshoe prior:
\begin{align*}
( \theta_i \mid \lambda_i, \tau ) &\sim \Nor \left ( 0 , \lambda_i^2 \right ), ( \lambda_i \mid \tau ) \sim C^+ \left ( 0 , \tau \right ), \tau \sim C^+ \left ( 0 , \eta \right ), \lambda_i, \tau > 0, \eta>0.
\end{align*}
To show that the hyper-parameter $\eta$ has minimal effect on inference, we study the performance of the horseshoe prior for the Efron's sum-of-squares problem with a sparse $\theta$ with a single non-zero component of magnitude $10$ and compare the posterior summary for three different values of $\eta \in \{0.5,1,5\}$. The observed value of $\sum_{i=1}^{n} y_i^2$ is $199{\cdot}3$. As Table \ref{tab:compare-tau-summary} and Fig. \ref{fig:compare-tau-density} show, the posterior density under the three different choices of scale parameters are very similar, as are the posterior mean and quantiles, with the closest answer being given by the choice of $\eta = 1$. This supports our choice of unit scale for the global-shrinkage parameter $\tau$.
\begin{figure}[ht!]%
\centering
\includegraphics[width = 0.5\linewidth, height = 2in]{compare-tau-3-scales.eps}%
\caption{Posterior densities of the parameter of interest $\psi = \sum_{i=1}^{p} \theta_i^2$, for the horseshoe prior with $\eta\in\{0.5, 1, 10\}$. The horizontal is at the true value $\psi = 100$. }%
\label{fig:compare-tau-density}%
\end{figure}
\begin{table}[!h]
\centering
\def~{\hphantom{0}}
\caption{Summary statistics and standard deviation of the posterior distribution under the horseshoe prior for $\eta \in \{0.5,1,5\}$ for the sum-of-squares problem. The true $\theta$ is a $100$-dimensional sparse vector with one non-zero component with magnitude $\theta_1 = 10$ and $\sum_{i=1}^{100} y_i^2 = 199.3$.}{%
{\footnotesize
\begin{tabular}{rrrrrrrr}
& Minimum & $Q_1$ & Median & Mean & $Q_3$ & Max & Std. Dev. \\
$\eta$ = 0${\cdot}$5 & 40${\cdot}$38 & 88${\cdot}$3 & 102${\cdot}$2 & 103${\cdot}$5 & 117${\cdot}$2 & 173${\cdot}$1 & 21${\cdot}$4 \\
$\eta = 1$ & 48${\cdot}$64 & 87${\cdot}$67 & 100${\cdot}$7 & 101${\cdot}$8 & 115${\cdot}$1 & 180${\cdot}$5 & 21${\cdot}$4 \\
$\eta = 5$ & 49${\cdot}$45 & 88${\cdot}$3 & 101${\cdot}$1 & 102${\cdot}$5 & 115${\cdot}$6 & 177${\cdot}$3 & 21${\cdot}$1 \\
\end{tabular}%
}%
}
\label{tab:compare-tau-summary}
%\begin{tabnote}
%HS, Horseshoe.
%\end{tabnote}
\end{table}
\section{Connections with the Penalized Complexity Priors}
The penalized complexity (PC) prior of \citet{simpson2014pc} on the precision parameter $\tau$ is given in Equation (3.3) of their article
$$
p(\tau) = \frac{\lambda}{2} \tau^{-3/2} \exp(-\lambda \tau^{-1/2}), \tau >0,
$$
where the parameter $\theta$ is a normal scale mixture, defined as, $\theta\mid\tau \sim \Nor (0, \tau^{-1} R^{-1})$ for some positive semi-definite $R$ and the data distribution can be taken as $y\mid \theta\sim \Nor(0, \sigma^2\mathrm{I})$. Without loss of generality, let $R$ be the identity matrix. A change of variables $\sigma^2 = \tau^{-1}$ yields the prior on the variance
$$
p(\sigma^2) = \frac{\lambda}{2} (\sigma^2)^{-1/2} \exp\{-\lambda (\sigma^2)^{1/2}\}, \sigma^2 >0.
$$
Thus, $p(\sigma^2) \sim (\sigma^2)^{-1/2}$ when $\sigma^2$ is close to zero and $p(\sigma^2) \sim \exp(-\lambda (\sigma^2)^{1/2})$ when $\sigma^2$ is large. This shows the PC prior on $\sigma^2$ has an unbounded mass at the origin, and Reviewer 1 is correct that this is an improvement over the inverse gamma prior on $\sigma^2$. However, the prior on $\sigma^2$ still has an exponentially decaying tail. As a result, the marginal PC prior on $\theta$ should behave similarly to the horseshoe prior on $\theta$ around the origin. It will, however, shrink the large signals due to the exponential tails of $\sigma^2$. Contrast this to the horseshoe priors on $\theta$ that results in both an unbounded mass at the origin as well as polynomially decaying tails for $\theta$ \citep[Fig.~1 of][]{carvalho2010horseshoe}. This distinction may not matter much in hypothesis testing problems, where separating zero versus non-zero is the major aim, but an application of Theorem 2 of \citet{carvalho2010horseshoe} shows the performance of PC priors in estimating large signals will be sub-optimal. In particular, the regularly-varying tails of the horseshoe priors on the variance parameter that results in the regularly varying tails of the marginals of $\theta$ (see Theorem 2.1 of the revised manuscript) does not appear to be applicable to the PC priors.
\citet{simpson2014pc} admit to this behavior of the PC prior in several places of their manuscript. On page 18, after Theorem 3 they say ``This result suggests the PC prior will shrink strongly, \ldots, due to the relatively light tail of the exponential.'' They also devote Section 4.5 of the article to point out the unsuitability of the PC priors as sparsity priors and concludes the section by noting ``This is the only situation we encountered in which the exponential tails of the PC priors are problematic.'' Thus, although we have not compared it's simulations, it appears the PC priors were designed with a different goal in mind from the problems we consider here.
}
\bibliographystyle{biometrika}
\bibliography{horseshoe-plus}
\end{document}
%
% Math commands by Thomas Minka
%
% Revised by Jyotishka Datta & Brandon Willard
%
% TODO: should/could we put includes for dependencies in here?
%
\newcommand{\var}{{\rm var}}
\newcommand{\Tr}{^{\rm T}}
\newcommand{\rmlog}{\rm log}
\newcommand{\vtrans}[2]{{#1}^{(#2)}}
\newcommand{\kron}{\otimes}
\newcommand{\schur}[2]{({#1} | {#2})}
\newcommand{\schurdet}[2]{\left| ({#1} | {#2}) \right|}
\newcommand{\had}{\circ}
\newcommand{\diag}{{\rm diag}}
\newcommand{\invdiag}{\diag^{-1}}
\newcommand{\rank}{{\rm rank}}
% careful: ``null'' is already a latex command
\newcommand{\nullsp}{{\rm null}}
\newcommand{\tr}{{\rm tr}}
\renewcommand{\vec}{{\rm vec}}
\newcommand{\vech}{{\rm vech}}
\renewcommand{\det}[1]{\left| #1 \right|}
\newcommand{\pdet}[1]{\left| #1 \right|_{+}}
\newcommand{\pinv}[1]{#1^{+}}
\newcommand{\erf}{{\rm erf}}
\newcommand{\hypergeom}[2]{{}_{#1}F_{#2}}
% boldface characters
\renewcommand{\a}{{\bf a}}
\renewcommand{\b}{{\bf b}}
\renewcommand{\c}{{\bf c}}
\renewcommand{\d}{{\rm d}} % for derivatives
\newcommand{\e}{{\bf e}}
\newcommand{\f}{{\bf f}}
\newcommand{\g}{{\bf g}}
\newcommand{\h}{{\bf h}}
%\newcommand{\k}{{\bf k}}
% in Latex2e this must be renewcommand
\renewcommand{\k}{{\bf k}}
\newcommand{\m}{{\bf m}}
\newcommand{\n}{{\bf n}}
%\renewcommand{\o}{{\bf o}}
\newcommand{\p}{{\bf p}}
%\newcommand{\q}{{\bf q}}
\renewcommand{\r}{{\bf r}}
\newcommand{\s}{{\bf s}}
\renewcommand{\t}{{\bf t}}
\renewcommand{\u}{{\bf u}}
\renewcommand{\v}{{\bf v}}
\newcommand{\w}{{\bf w}}
%\newcommand{\x}{{\bf x}}
\newcommand{\y}{{\bf y}}
%\newcommand{\z}{{\bf z}}
\newcommand{\A}{{\bf A}}
\newcommand{\B}{{\bf B}}
%\newcommand{\C}{{\bf C}}
\newcommand{\D}{{\bf D}}
\newcommand{\E}{{\bf E}}
\newcommand{\F}{{\bf F}}
%\newcommand{\G}{{\bf G}}
\renewcommand{\H}{{\bf H}}
\newcommand{\I}{{\bf I}}
\newcommand{\J}{{\bf J}}
\newcommand{\K}{{\bf K}}
\renewcommand{\L}{{\bf L}}
\newcommand{\M}{{\bf M}}
\newcommand{\Nor}{{\cal N}} % for normal density
%\newcommand{\N}{{\bf N}}
\renewcommand{\O}{{\bf O}}
\renewcommand{\P}{{\bf P}}
\newcommand{\Q}{{\bf Q}}
\newcommand{\R}{{\bf R}}
\renewcommand{\S}{{\bf S}}
%\newcommand{\T}{{\bf T}}
%\newcommand{\U}{{\bf U}}
\newcommand{\V}{{\bf V}}
\newcommand{\W}{{\bf W}}
\newcommand{\X}{{\bf X}}
\newcommand{\Y}{{\bf Y}}
\newcommand{\Z}{{\bf Z}}
% this is for latex 2.09
% unfortunately, the result is slanted - use Latex2e instead
%\newcommand{\bfLambda}{\mbox{\boldmath$\Lambda$}}
% this is for Latex2e
\newcommand{\bfLambda}{\boldsymbol{\Lambda}}
% Yuan Qi's boldsymbol
\newcommand{\bsigma}{\boldsymbol{\sigma}}
\newcommand{\balpha}{\boldsymbol{\alpha}}
\newcommand{\bpsi}{\boldsymbol{\psi}}
\newcommand{\bphi}{\boldsymbol{\phi}}
\newcommand{\bbeta}{\boldsymbol{\beta}}
%\newcommand{\Beta}{\boldsymbol{\eta}}
\newcommand{\btau}{\boldsymbol{\tau}}
\newcommand{\bvarphi}{\boldsymbol{\varphi}}
\newcommand{\bzeta}{\boldsymbol{\zeta}}
\newcommand{\bnabla}{\boldsymbol{\nabla}}
\newcommand{\blambda}{\boldsymbol{\lambda}}
\newcommand{\bLambda}{\mathbf{\Lambda}}
\newcommand{\btheta}{\boldsymbol{\theta}}
\newcommand{\bpi}{\boldsymbol{\pi}}
\newcommand{\bPi}{\boldsymbol{\Pi}}
\newcommand{\bxi}{\boldsymbol{\xi}}
\newcommand{\bSigma}{\boldsymbol{\Sigma}}
\newcommand{\bgamma}{\mathbf{\gamma}}
\newcommand{\bGamma}{\mathbf{\Gamma}}
\newcommand{\bmu}{\boldsymbol{\mu}}
\newcommand{\bnu}{\boldsymbol{\nu}}
\newcommand{\bPsi}{\mathbf{\Psi}}
\newcommand{\bepsilon}{\boldsymbol{\epsilon}}
\newcommand{\bOmega}{\boldsymbol{\Omega}}
\newcommand{\1}{{\bf 1}}
\newcommand{\0}{{\bf 0}}
%\newcommand{\comment}[1]{}
\newcommand{\bs}{\backslash}
\newcommand{\ben}{\begin{enumerate}}
\newcommand{\een}{\end{enumerate}}
\newcommand{\beq}{\begin{equation}}
\newcommand{\eeq}{\end{equation}}
\newcommand{\notS}{{\backslash S}}
\newcommand{\nots}{{\backslash s}}
\newcommand{\noti}{{\backslash i}}
\newcommand{\notj}{{\backslash j}}
\newcommand{\nott}{\backslash t}
\newcommand{\notone}{{\backslash 1}}
\newcommand{\nottp}{\backslash t+1}
% \newcommand{\notz}{\backslash z}
\newcommand{\notk}{{^{\backslash k}}}
%\newcommand{\noti}{{^{\backslash i}}}
\newcommand{\notij}{{^{\backslash i,j}}}
\newcommand{\notg}{{^{\backslash g}}}
\newcommand{\wnoti}{{_{\w}^{\backslash i}}}
\newcommand{\wnotg}{{_{\w}^{\backslash g}}}
\newcommand{\vnotij}{{_{\v}^{\backslash i,j}}}
\newcommand{\vnotg}{{_{\v}^{\backslash g}}}
\newcommand{\half}{\frac{1}{2}}
\newcommand{\quart}{\frac{1}{4}}
\newcommand{\msgb}{m_{t \leftarrow t+1}}
\newcommand{\msgf}{m_{t \rightarrow t+1}}
\newcommand{\msgfp}{m_{t-1 \rightarrow t}}
\newcommand{\proj}[1]{{\rm proj}\negmedspace\left[#1\right]}
\newcommand{\argmin}{\operatornamewithlimits{argmin}}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}
\newcommand{\dif}{\mathrm{d}}
\newcommand{\abs}[1]{\lvert#1\rvert}
%\newcommand{\norm}[1]{\lVert#1\rVert}
\newcommand{\vectornorm}[1]{\left|\left|#1\right|\right|}
\newcommand{\Norm}{\mathcal{N}}
\newcommand{\bx}{{\bf x}}
\newcommand{\ba}{{\bf a}}
\newcommand{\bb}{{\bf b}}
\newcommand{\bc}{{\bf c}}
\newcommand{\bd}{{\bf d}}
\newcommand{\bX}{{\bf X}}
\newcommand{\by}{{\bf y}}
\newcommand{\IG}{\mathcal{IG}}
\newcommand{\dd}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\lhat}[1][i]{\hat\lambda_{#1}^{-1(g)}}
\newcommand{\what}[1][j]{\hat\omega_{#1}^{-1(g)}}
\newcommand{\bone}{{\bf 1}}
\newcommand{\Li}{\hat\Lambda^{-1(g)}}
\newcommand{\Oi}{\hat\Omega^{-1(g)}}
\newcommand{\iid}{\stackrel{\mathrm{iid}}{\sim}}
\newcommand{\iidp}{\stackrel{\mathrm{P}}{=}}
\newcommand{\iidd}{\stackrel{\mathrm{D}}{=}}
\newcommand{\defeq}{\operatorname{:=}}
% the last {} is a hack for double subscript errors
\newcommand{\estHsp}{\ensuremath{{\hat{\theta}}_{HS+}}{}}
\newcommand{\estHs}{\ensuremath{{\hat{\theta}}_{HS}}{}}
\newcommand{\estJs}{\ensuremath{{\hat{\theta}}_{JS}}{}}
\newcommand{\MSE}{\mathrm{MSE}}
%
% Meijer-G additions
%
%\DeclarePairedDelimiterX\MeijerM[3]{\lparen}{\rparen}%
%{\begin{smallmatrix}#1 \\ #2\end{smallmatrix}\delimsize\vert\,#3}
%
%\newcommand\MeijerG[8][]{%
% G^{\,#2,#3}_{#4,#5}\MeijerM[#1]{#6}{#7}{#8}}
%
%\WithSuffix\newcommand\MeijerG*[7]{G^{\,#1,#2}_{#3,#4}\MeijerM*{#5}{#6}{#7}}
%% end Meijer-G
%
%%
%% Generalized Hypergeometric Function (pFq)
%%
%\DeclarePairedDelimiterX\pFqM[3]{\lparen}{\rparen}%
%{\begin{smallmatrix}#1 \\ #2\end{smallmatrix}\delimsize\vert\,#3}
%
%\newcommand\pFq[6][]{%
% {}_{#2}F_{#3}\pFqM[#1]{#4}{#5}{#6}}
%\WithSuffix\newcommand\pFq*[5]{{}_{#1}F_{#2}\pFqM*{#3}{#4}{#5}}
% end pFq
\newtheorem{theorem}{Theorem}
\numberwithin{theorem}{section}
\newtheorem{Proof}{Proof}
\newtheorem{Def}{Definition}
\numberwithin{Def}{section}
\newtheorem{remark}{Remark}
\numberwithin{remark}{section}
\newtheorem{Qes}{Question}
\newtheorem{proposition}{Proposition}
\numberwithin{proposition}{section}
\newtheorem{lemma}{Lemma}
\numberwithin{lemma}{section}
\newtheorem{Cor}{Corollary}
\numberwithin{Cor}{section}
\newtheorem{Exa}{Example}
\newtheorem{Eq}{Equation}
\newtheorem{assn}{Assumption}
\newtheorem{result}[theorem]{Result}
%\usepackage{booktabs,array}
%\def\Midrule{\midrule[\heavyrulewidth]}
%\newcount\rowc
%
%\makeatletter
%\def\ttabular{%
%\hbox\bgroup
%\let\\\cr
%\def\rulea{\ifnum\rowc=\@ne \hrule height 1.0pt \fi}
%\def\ruleb{
%\ifnum\rowc=1\hrule height 1.0pt \else
%\ifnum\rowc=3\hrule height 0.0pt%\heavyrulewidth
%\ifnum\rowc= 3 \hrule height 0.5pt \else%\heavyrulewidth
%\ifnum\rowc= 5 \hrule height 0.5pt \else%\heavyrulewidth
%\ifnum\rowc= 7 \hrule height 0.5pt \else%\heavyrulewidth
%\ifnum\rowc= 9 \hrule height 0.5pt \else%\heavyrulewidth
%\ifnum\rowc= 11 \hrule height 0.5pt %\heavyrulewidth
%\else \hrule height 0pt%\lightrulewidth
%\fi\fi\fi\fi\fi\fi}
%\valign\bgroup
%\global\rowc\@ne
%\rulea
%\hbox to 7em{\strut \hfill##\hfill}%
%\ruleb
%&&%
%\global\advance\rowc\@ne
%\hbox to 7em{\strut\hfill##\hfill}%
%\ruleb
%\cr}
%\def\endttabular{%
%\crcr\egroup\egroup}
\begin{thebibliography}{49}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Andrade \& Omey(2011)}]{andrade2011bayesian-b}
\textsc{Andrade, J.} \& \textsc{Omey, E.} (2011).
\newblock {B}ayesian robustness modelling using subexponential distributions
and related classes of distributions.
\newblock Tech. rep., Katholieke Universiteit Leuven.
\bibitem[{Andrade \& O'Hagan(2006)}]{andrade2006bayesian}
\textsc{Andrade, J. A.~A.} \& \textsc{O'Hagan, A.} (2006).
\newblock {B}ayesian robustness modeling using regularly varying distributions.
\newblock \textit{{B}ayesian Analysis} \textbf{1}, 169--188.
\bibitem[{Andrade \& O'Hagan(2011)}]{andrade2011bayesian}
\textsc{Andrade, J. A.~A.} \& \textsc{O'Hagan, A.} (2011).
\newblock {B}ayesian robustness modelling of location and scale parameters.
\newblock \textit{Scandinavian Journal of Statistics} \textbf{38}, 691--711.
\bibitem[{Barndorff-Nielsen et~al.(1982)Barndorff-Nielsen, Kent \&
S{\o}rensen}]{barndorff1982normal}
\textsc{Barndorff-Nielsen, O.~E.}, \textsc{Kent, J.~T.} \& \textsc{S{\o}rensen,
M.} (1982).
\newblock Normal variance-mean mixtures and z distributions.
\newblock \textit{International Statistical Review/Revue Internationale de
Statistique} \textbf{50}, 145--159.
\bibitem[{Berger \& Bernardo(1989)}]{berger1989estimating}
\textsc{Berger, J.~O.} \& \textsc{Bernardo, J.~M.} (1989).
\newblock Estimating a product of means: {B}ayesian analysis with reference
priors.
\newblock \textit{Journal of the American Statistical Association} \textbf{84},
200--207.
\bibitem[{Berger \& Bernardo(1992)}]{berger1992development}
\textsc{Berger, J.~O.} \& \textsc{Bernardo, J.~M.} (1992).
\newblock On the development of reference priors.
\newblock \textit{{B}ayesian Statistics} \textbf{4}, 35--60.
\bibitem[{Berger et~al.(1999)Berger, Liseo \& Wolpert}]{berger1999}
\textsc{Berger, J.~O.}, \textsc{Liseo, B.} \& \textsc{Wolpert, R.~L.} (1999).
\newblock Integrated likelihood methods for eliminating nuisance parameters.
\newblock \textit{Statistical Science} \textbf{14}, 1--28.
\bibitem[{Berger et~al.(1998)Berger, Philippe \& Robert}]{berger1998estimation}
\textsc{Berger, J.~O.}, \textsc{Philippe, A.} \& \textsc{Robert, C.~P.} (1998).
\newblock Estimation of quadratic functions: noninformative priors for
non-centrality parameters.
\newblock \textit{Statistica Sinica} \textbf{8}, 359--375.
\bibitem[{Bernardo(1979)}]{bernardo1979reference}
\textsc{Bernardo, J.~M.} (1979).
\newblock {Reference posterior distributions for {B}ayesian inference (with
discussion)}.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{41}, 113--147.
\bibitem[{Bhadra et~al.(2015)Bhadra, Datta, Polson \&
Willard}]{bhadra2015horseshoe+}
\textsc{Bhadra, A.}, \textsc{Datta, J.}, \textsc{Polson, N.~G.} \&
\textsc{Willard, B.} (2015).
\newblock The horseshoe+ estimator of ultra-sparse signals.
\newblock \textit{arXiv preprint arXiv:1502.00560} .
\bibitem[{Bingham et~al.(1989)Bingham, Goldie \& Teugels}]{bingham1989regular}
\textsc{Bingham, N.~H.}, \textsc{Goldie, C.~M.} \& \textsc{Teugels, J.~L.}
(1989).
\newblock \textit{Regular {V}ariation}, vol.~27 of \textit{Encyclopedia of
Mathematics and its Applications}.
\newblock Cambridge University Press, Cambridge.
\bibitem[{Box \& Tiao(1962)}]{box1962further}
\textsc{Box, G.~E.} \& \textsc{Tiao, G.~C.} (1962).
\newblock A further look at robustness via {B}ayes's theorem.
\newblock \textit{Biometrika} \textbf{49}, 419--432.
\bibitem[{Carvalho et~al.(2010)Carvalho, Polson \&
Scott}]{carvalho2010horseshoe}
\textsc{Carvalho, C.~M.}, \textsc{Polson, N.~G.} \& \textsc{Scott, J.~G.}
(2010).
\newblock The horseshoe estimator for sparse signals.
\newblock \textit{Biometrika} \textbf{97}, 465--480.
\bibitem[{Creasy(1954)}]{creasy1954limits}
\textsc{Creasy, M.~A.} (1954).
\newblock Symposium on interval estimation: {L}imits for the ratio of means.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{16}, 186--194.
\bibitem[{Datta \& Ghosh(2013)}]{datta2013asymptotic}
\textsc{Datta, J.} \& \textsc{Ghosh, J.~K.} (2013).
\newblock {Asymptotic properties of {B}ayes risk for the horseshoe prior}.
\newblock \textit{{B}ayesian Analysis} \textbf{8}, 111--132.
\bibitem[{Dawid(1973)}]{dawid1973posterior}
\textsc{Dawid, A.~P.} (1973).
\newblock Posterior expectations for large observations.
\newblock \textit{Biometrika} \textbf{60}, 664--667.
\bibitem[{Dawid et~al.(1973)Dawid, Stone \& Zidek}]{dawid1973marginalization}
\textsc{Dawid, A.~P.}, \textsc{Stone, M.} \& \textsc{Zidek, J.~V.} (1973).
\newblock Marginalization paradoxes in {B}ayesian and structural inference
(with discussion).
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{35}, 189--233.
\bibitem[{de~Finetti(1961)}]{de1961bayesian}
\textsc{de~Finetti, B.} (1961).
\newblock The {B}ayesian approach to the rejection of outliers.
\newblock In \textit{Proc. 4th {B}erkeley {S}ympos. {M}ath. {S}tatist. and
{P}rob., {V}ol. {I}}. Univ. California Press, Berkeley, Calif.
\bibitem[{Efron(1973)}]{efron1973discussion}
\textsc{Efron, B.} (1973).
\newblock {Discussion of Marginalization paradoxes in {B}ayesian and structural
inference, by A. P. Dawid, M. Stone, and J. V. Zidek}.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{35}, 189--233.
\bibitem[{Efron(1978)}]{efron1978controversies}
\textsc{Efron, B.} (1978).
\newblock Controversies in the foundations of statistics.
\newblock \textit{American Mathematical Monthly} \textbf{85}, 231--246.
\bibitem[{Efron(2011)}]{efron2011tweedie}
\textsc{Efron, B.} (2011).
\newblock Tweedie's formula and selection bias.
\newblock \textit{Journal of the American Statistical Association}
\textbf{106}, 1602--1614.
\bibitem[{Feller(1971)}]{feller1971}
\textsc{Feller, W.} (1971).
\newblock \textit{An Introduction to Probability Theory and Its Applications,
Vol. 2}.
\newblock New York, NY: Wiley, 2nd ed.
\bibitem[{Fieller(1940)}]{fieller1940biological}
\textsc{Fieller, E.} (1940).
\newblock The biological standardization of insulin.
\newblock \textit{Supplement to the Journal of the Royal Statistical Society}
\textbf{7}, 1--64.
\bibitem[{Gelman(2006)}]{gelman2006prior}
\textsc{Gelman, A.} (2006).
\newblock Prior distributions for variance parameters in hierarchical models
(comment on article by {B}rowne and {D}raper).
\newblock \textit{{B}ayesian Analysis} \textbf{1}, 515--533.
\bibitem[{Ghosh et~al.(2015)Ghosh, Tang, Ghosh, Chakrabarti
et~al.}]{ghosh2015asymptotic}
\textsc{Ghosh, P.}, \textsc{Tang, X.}, \textsc{Ghosh, M.}, \textsc{Chakrabarti,
A.} et~al. (2015).
\newblock Asymptotic properties of {B}ayes risk of a general class of shrinkage
priors in multiple hypothesis testing under sparsity.
\newblock \textit{{B}ayesian Analysis} \textbf{to appear}, 1--44.
\bibitem[{Gneiting(1997)}]{gneiting1997normal}
\textsc{Gneiting, T.} (1997).
\newblock Normal scale mixtures and dual probability densities.
\newblock \textit{Journal of Statistical Computation and Simulation}
\textbf{59}, 375--384.
\bibitem[{Good(1995)}]{good1995c439}
\textsc{Good, I.} (1995).
\newblock {C439. Dual density functions}.
\newblock \textit{Journal of Statistical Computation and Simulation}
\textbf{52}, 193--194.
\bibitem[{Irony \& Singpurwalla(1997)}]{irony1997non}
\textsc{Irony, T.~Z.} \& \textsc{Singpurwalla, N.~D.} (1997).
\newblock Non-informative priors do not exist: {A} discussion with {J}os{\'e}
{M}. {B}ernardo.
\newblock \textit{Journal of Statistical Planning and Inference} \textbf{65},
159--177.
\bibitem[{Jeffreys(1939)}]{Jeffreys1961}
\textsc{Jeffreys, H.} (1939).
\newblock \textit{Theory of {P}robability}.
\newblock Oxford University Press, Oxford.
\bibitem[{Lindley(1968)}]{lindley1968choice}
\textsc{Lindley, D.~V.} (1968).
\newblock The choice of variables in multiple regression.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{30}, 31--66.
\bibitem[{Liseo(1993)}]{liseo1993elimination}
\textsc{Liseo, B.} (1993).
\newblock Elimination of nuisance parameters with reference priors.
\newblock \textit{Biometrika} \textbf{80}, 295--304.
\bibitem[{Nadarajah(2009)}]{nadarajah2009pdfs}
\textsc{Nadarajah, S.} (2009).
\newblock Pdfs and dual pdfs.
\newblock \textit{The American Statistician} \textbf{63}, 45--48.
\bibitem[{O'Hagan(1979)}]{o1979outliers}
\textsc{O'Hagan, A.} (1979).
\newblock On outlier rejection phenomena in {B}ayes inference.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{41}, 358--367.
\bibitem[{O'Hagan(1990)}]{o1990outliers}
\textsc{O'Hagan, A.} (1990).
\newblock Outliers and credence for location parameter inference.
\newblock \textit{Journal of the American Statistical Association} \textbf{85},
172--176.
\bibitem[{O'Hagan \& Pericchi(2012)}]{o2012bayesian}
\textsc{O'Hagan, A.} \& \textsc{Pericchi, L.} (2012).
\newblock {B}ayesian heavy-tailed models and conflict resolution: A review.
\newblock \textit{Brazilian Journal of Probability and Statistics} \textbf{26},
372--401.
\bibitem[{Olver et~al.(2010)Olver, Lozier, Boisvert \& Clark}]{Olver:2010:NHMF}
\textsc{Olver, F.~W.~J.}, \textsc{Lozier, D.~W.}, \textsc{Boisvert, R.~F.} \&
\textsc{Clark, C.~W.}, eds. (2010).
\newblock \textit{{NIST Handbook of Mathematical Functions}}.
\newblock New York, NY: Cambridge University Press.
\newblock Print companion to NIST:DLMF.
\bibitem[{Pericchi \& Smith(1992)}]{pericchi1992exact}
\textsc{Pericchi, L.} \& \textsc{Smith, A.} (1992).
\newblock Exact and approximate posterior moments for a normal location
parameter.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{54}, 793--804.
\bibitem[{Polson \& Scott(2010)}]{polson2010shrink}
\textsc{Polson, N.~G.} \& \textsc{Scott, J.~G.} (2010).
\newblock Shrink globally, act locally: sparse {B}ayesian regularization and
prediction.
\newblock \textit{{B}ayesian Statistics} \textbf{9}, 501--538.
\bibitem[{Polson \& Scott(2012{\natexlab{a}})}]{polson2012local}
\textsc{Polson, N.~G.} \& \textsc{Scott, J.~G.} (2012{\natexlab{a}}).
\newblock {Local shrinkage rules, L{\'e}vy processes and regularized
regression}.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{74}, 287--311.
\bibitem[{Polson \& Scott(2012{\natexlab{b}})}]{polson2012half}
\textsc{Polson, N.~G.} \& \textsc{Scott, J.~G.} (2012{\natexlab{b}}).
\newblock On the half-{C}auchy prior for a global scale parameter.
\newblock \textit{{B}ayesian Analysis} \textbf{7}, 887--902.
\bibitem[{Rubio \& Steel(2014)}]{rubio2014inference}
\textsc{Rubio, F.~J.} \& \textsc{Steel, M.~F.} (2014).
\newblock Inference in two-piece location-scale models with {J}effreys priors.
\newblock \textit{{B}ayesian Analysis} \textbf{9}, 1--22.
\bibitem[{Scott(2014)}]{scott2014comment}
\textsc{Scott, J.~G.} (2014).
\newblock {Comment on Article by Rubio and Steel}.
\newblock \textit{{B}ayesian Analysis} \textbf{9}, 25--28.
\bibitem[{Scott \& Berger(2006)}]{scott2006exploration}
\textsc{Scott, J.~G.} \& \textsc{Berger, J.~O.} (2006).
\newblock An exploration of aspects of {B}ayesian multiple testing.
\newblock \textit{Journal of Statistical Planning and Inference} \textbf{136},
2144--2162.
\bibitem[{{Simpson} et~al.(2014){Simpson}, {Rue}, {Martins}, {Riebler} \&
{S{\o}rbye}}]{simpson2014pc}
\textsc{{Simpson}, D.~P.}, \textsc{{Rue}, H.}, \textsc{{Martins}, T.~G.},
\textsc{{Riebler}, A.} \& \textsc{{S{\o}rbye}, S.~H.} (2014).
\newblock {Penalising model component complexity: A principled, practical
approach to constructing priors}.
\newblock \textit{ArXiv e-prints} .
\bibitem[{{Stan Development Team}(2014)}]{stan-software:2014}
\textsc{{Stan Development Team}} (2014).
\newblock {Stan: A C++ Library for Probability and Sampling, Version 2.2}.
\bibitem[{Sun \& Ye(1999)}]{sun1999reference}
\textsc{Sun, D.} \& \textsc{Ye, K.} (1999).
\newblock Reference priors for a product of normal means when variances are
unknown.
\newblock \textit{The Canadian Journal of Statistics} \textbf{27}, 97--103.
\bibitem[{Sun \& Ye(1995)}]{sun1995reference}
\textsc{Sun, D.~C.} \& \textsc{Ye, K.} (1995).
\newblock Reference prior {B}ayesian analysis for normal mean products.
\newblock \textit{Journal of the American Statistical Association} \textbf{90},
589--597.
\bibitem[{van~der Pas et~al.(2014)van~der Pas, Kleijn \& van~der
Vaart}]{van2014horseshoe}
\textsc{van~der Pas, S.}, \textsc{Kleijn, B.} \& \textsc{van~der Vaart, A.}
(2014).
\newblock The horseshoe estimator: Posterior concentration around nearly black
vectors.
\newblock \textit{Electronic Journal of Statistics} \textbf{8}, 2585--2618.
\bibitem[{West(1984)}]{west84}
\textsc{West, M.} (1984).
\newblock Outlier models and prior distributions in {B}ayesian linear
regression.
\newblock \textit{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} \textbf{46}, 431--439.
\end{thebibliography}