Tractable Bayesian Density Regression via Logit Stick-Breaking Priors

Extracted source text · arXiv 1701.02969

%%%%%%%%%%%%%%%%%%%%%%% file template.tex %%%%%%%%%%%%%%%%%%%%%%%%%
%
% This is a general template file for the LaTeX package SVJour3
% for Springer journals.          Springer Heidelberg 2010/09/16
%
% Copy it to a new file with a new name and use it as the basis
% for your article. Delete % signs as needed.
%
% This template includes a few options for different layouts and
% content for various journals. Please consult a previous issue of
% your journal as needed.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% First comes an example EPS file -- just ignore it and
% proceed on the \documentclass line
% your LaTeX will extract the file if required
\begin{filecontents*}{example.eps}
%!PS-Adobe-3.0 EPSF-3.0
%%BoundingBox: 19 19 221 221
%%CreationDate: Mon Sep 29 1997
%%Creator: programmed by hand (JK)
%%EndComments
gsave
newpath
  20 20 moveto
  20 220 lineto
  220 220 lineto
  220 20 lineto
closepath
2 setlinewidth
gsave
  .4 setgray fill
grestore
stroke
grestore
\end{filecontents*}
%
\RequirePackage{fix-cm}
%
%\documentclass{svjour3}                     % onecolumn (standard format)
%\documentclass[smallcondensed]{svjour3}     % onecolumn (ditto)
%\documentclass[smallextended]{svjour3}       % onecolumn (second format)
\documentclass[twocolumn]{svjour3}          % twocolumn
%
\smartqed  % flush right qed marks, e.g. at end of proof
%
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{scrextend}
\usepackage[ruled,norelsize,vlined]{algorithm2e}
\usepackage[colorlinks,citecolor=blue,urlcolor=blue,filecolor=blue,backref=page]{hyperref}
\usepackage{natbib}
\usepackage{bm}
\usepackage{bbm}
\usepackage{subfigure}
\usepackage{hyperref}
\renewcommand{\baselinestretch}{1.4} 


\usepackage{tikz}
\usetikzlibrary{fit,positioning,calc}

\usepackage{arydshln}

\setlength\dashlinedash{12pt}
\setlength\dashlinegap{8pt}
\setlength\arrayrulewidth{0.2pt}

%
% \usepackage{mathptmx}      % use Times fonts if available on your TeX system
%
% insert here the call for the packages your document requires
%\usepackage{latexsym}
% etc.
%
% please place your own definitions here and don't use \def but
% \newcommand{}{}
%
% Insert the name of "your journal" with
% \journalname{myjournal}
%
\def\Bka{{\it Biometrika}}
\def\AIC{\textsc{aic}}
\def\intercal{{ \mathrm{\scriptscriptstyle T} }}
\def\T{{ \mathrm{\scriptscriptstyle T} }}
\def\v{{\varepsilon}}
\newcommand{\deltain}{\rotatebox[origin=c]{180}{$\boldsymbol{\Delta}$}}

\newcommand{\I}{{1}}

\begin{document}

\title{Tractable Bayesian density regression \\ via logit stick--breaking priors%\thanks{Grants or other notes
%about the article that should go on the front page should be
%placed here. General acknowledgments should be placed at the end of the article.}
}
%\subtitle{Do you have a subtitle?\\ If so, write it here}

\titlerunning{Tractable Bayesian density regression via logit stick--breaking priors}        % if too long for running head

\author{Tommaso Rigon \and
Daniele Durante         
}

\authorrunning{Rigon \and Durante} % if too long for running head

\institute{ Tommaso Rigon \at
              Department of Decision Sciences, Bocconi University   \\
              \email{tommaso.rigon@phd.unibocconi.it}
              \and
Daniele Durante \at
              Department of Decision Sciences, Bocconi University   \\
               \email{daniele.durante@unibocconi.it}
}

\date{Received: date / Accepted: date}
% The correct dates will be entered by the editor


\maketitle


\begin{abstract}
There is an increasing interest in learning how the distribution of a response variable changes with a set of predictors.  Bayesian nonparametric dependent mixture models provide a useful approach to flexibly address this goal. However, many formulations are characterized by difficult interpretation and intractable computational methods. Motivated by these issues, we define a class of predictor--dependent infinite mixture models, which relies on a formal representation of the stick--breaking construction via a continuation--ratio logistic regression, within an exponential family representation. This formulation maintains the same desirable properties of popular predictor--dependent stick-breaking priors, but leverages a recent P\'olya-Gamma data augmentation to facilitate tractable inference under a broader variety of routine--use computational methods. These methods include Markov Chain Monte Carlo via Gibbs sampling, Expectation Maximization algorithms, and a variational Bayes routine for scalable inference. The algorithms associated with these methods are tested in a toxicology study.
\keywords{Continuation--ratio logistic regression  \and Density regression \and Gibbs sampling \and  Expectation--maximization \and P\'olya-gamma \and Variational Bayes}
\end{abstract}

\section{Introduction}
\label{sec1}
There is a growing interest in density regression methods which allow the entire distribution of a univariate response variable $y \in \mathcal{Y}$ to be unknown, and changing with a vector of predictors $\bf{x}  \in \mathcal{X}$. Indeed, the increased flexibility provided by these procedures allows relevant improvements in inference and prediction compared to classical regression frameworks, as seen in several  applications \citep[e.g.][]{DP08, GRI11, WDPT14, GUT16}.

Within a Bayesian nonparametric framework, there is a wide set of alternative methodologies to provide flexible inference for conditional distributions. Most of these methods represent generalizations of the marginal density estimation problem for $f(y)$,  which is commonly addressed via Bayesian nonparametric mixture models of the form $f(y)= \int_{\boldsymbol{\Theta}} K(y ; \boldsymbol{\theta}) \mbox{d} P(  \boldsymbol{\theta})$, where $ K(y ; \boldsymbol{\theta})$ is a known parametric kernel indexed by $\boldsymbol{\theta} \in \boldsymbol{\Theta}$, and $P(  \boldsymbol{\theta})$ denotes an unknown mixing measure which is assigned a flexible prior $\Pi$. Popular choices for $\Pi$ are the Dirichlet process \citep{FE73}, the two-parameter Poisson--Dirichlet process \citep{PY97}, and other almost surely discrete random measures having a stick-breaking representation \citep{IJ01}. This choice leads to the infinite mixture model  
\begin{eqnarray}
f(y)= \int_{\boldsymbol{\Theta}} K(y ; \boldsymbol{\theta}) \mbox{d} P(  \boldsymbol{\theta})=\sum_{h=1}^{+\infty} \pi_h K(y ; \boldsymbol{\theta}_h), 
\label{mix_intro}
\end{eqnarray}
with $\pi_h=\nu_h \prod_{l=1}^{h-1}(1-\nu_l), \ h=1, \ldots, +\infty$. In \eqref{mix_intro}, the kernel parameters  $\boldsymbol{\theta}_h$, $h=1, \ldots, +\infty$ are distributed according to a base measure  $P_0$, whereas the stick--breaking weights $\nu_h \in (0,1)$, $h=1, \ldots, +\infty$, have independent $\mbox{Beta}(a_h,b_h)$ priors, so that $\sum_{h=1}^{+\infty}\pi_h=1$ almost surely.  Fixing $a_h=1$ and $b_h=a$ leads to a Dirichlet process mixture model, whereas the two-parameter Poisson--Dirichlet process mixture can be obtained letting $a_h=1-a$ and $b_h=b+ha$, $0 \leq a<1$ and $b>-a$. 

Model \eqref{mix_intro} has key computational benefits in allowing the implementation of simple Markov Chain Monte Carlo methods for posterior inference \citep[e.g.][]{EW95, NEA00}, and provides a consistent  procedure for density estimation \citep[e.g.][]{GHO99, TOK06, GHO07}.  This has motivated different generalizations of \eqref{mix_intro} to incorporate the conditional density inference problem for $f(y \mid {\bf{x}})=f_{{\bf x}}(y)$, by allowing the unknown random mixing measure $P_{\bf{x} }(\boldsymbol{\theta})$ to change with  $\bf{x}  \in \mathcal{X}$, under a dependent stick-breaking characterization \citep{MC99,MC00}. Popular representations consider predictor--independent mixing weights $\pi_h$, and incorporate changes with $\bf{x}  \in \mathcal{X}$ in the atoms $\boldsymbol{\theta}_h(\bf{x} )$ \citep[e.g.][]{DEI04,GEL05,CA06, CRU07}. Although these constructions have desirable theoretical properties \citep{BAR2012, PA013},  as noted in \cite{MC00} and \cite{GS06}, the predictor--independent assumption for the mixing weights can have limited flexibility in practice, thus requiring the introduction of many mixture components. This has motivated more general formulations allowing also  $\pi_h(\bf{x} )$, to change with the predictors. Relevant examples include the order-based dependent Dirichlet process \citep{GS06}, the kernel stick-breaking process \citep{DP08}, the infinite mixture models with predictor-dependent weights  \citep{AWW14}, and more recent representations for Bayesian dynamic inference \citep{GUT16}. These formulations provide a broader class of priors for Bayesian density regression, and have desirable theoretical properties \citep{BAR2012,PA013}. However their flexibility comes at a computational cost. In particular, the availability of simple algorithms for tractable posterior inference is limited by the specific construction and parameterization of these representations. 

The above issues motivate alternative formulations which preserve the theoretical properties, but facilitate tractable posterior computation under a broader variety of standard algorithms  in Bayesian inference, thus facilitating a wider implementation of nonparametric density regression. We address this goal  via a logit stick-breaking prior (\textsc{lsbp}) which relates each stick-breaking weight $\nu_h({\bf{x}} ) \in (0,1)$, to a function  $\eta_h(\bf{x} ) \in \Re$ of the covariates,  using the logistic link. The proposed statistical model is partially related to the probit stick-breaking prior of \cite{RD11}, which leverages the probit link and provides posterior inference via Markov Chain Monte Carlo (\textsc{mcmc}) algorithms. However, in large-scale applications---when either the number of observations or the dimension of the covariates is large---\textsc{mcmc} via Gibbs sampling could face scalability and mixing issues, thus motivating the development of  alternative computational routines. As we will outline in Section~\ref{Section3}, the logistic mapping allows simple posterior computation under a broader variety of routine--use algorithms beyond \textsc{mcmc}. These include a tractable Expectation Maximization (\textsc{em}) routine for point estimation, and simple variational Bayes (\textsc{vb}) for scalable posterior inference.  We shall emphasize that the overarching focus of our contribution is not on developing a novel methodological framework for Bayesian density regression, but on providing alternative representations within this well--established class of models which are characterized by improved interpretability and computational tractability for a wider set of  algorithms. To our knowledge this goal remains partially unaddressed, but represents a fundamental condition to facilitate routine implementation of Bayesian density regression by the practitioners. 

As discussed in Section  \ref{Section3}, there is also a formal connection between the \textsc{lsbp} and the  hierarchical mixtures of experts \citep{JJ94, BS03}, for which \textsc{vb} algorithms are available. \cite{RDC11} noticed a similar connection in their logistic stick-breaking process. However, their focus is exclusively on  clustering  of spatio--temporal data, and inference is only available via local \textsc{vb} routines based on the bound of \citet{JJ00}. Compared to the approaches mentioned above, our contribution is instead designed for a general class of density regression problems and provides additional algorithms---i.e. Gibbs sampling, \textsc{em} and a global \textsc{vb}---combining  the recent P\'olya-Gamma data augmentation for Bayesian logistic regression \citep{PS13}, and a continuation--ratio representation \citep{TU91} of the  \textsc{lsbp} which is described in Section~\ref{Section2}. The three routines are empirically compared in Section~\ref{Section4} using a real data toxicology study, previously considered in \cite{DP08}.  Section~\ref{Section5} provides concluding remarks.


\section{The logit stick-breaking prior (LSBP)}
\label{Section2}
This section presents a formal construction of the  \textsc{lsbp} via a continuation--ratio parameterization  of the hierarchical mechanism assigning the units to mixture components. As a natural extension of model~\eqref{mix_intro}, we consider the general class of predictor--dependent infinite mixture models
\begin{equation}
f_{\bf x}(y) =\int_{\boldsymbol{\Theta}} K_{{\bf x}}(y ; \boldsymbol{\theta}) \mbox{d} P_{\bf{x}}(  \boldsymbol{\theta})   =\sum_{h=1}^{+\infty} \pi_h({\bf x}) K_{\bf x}(y ; \boldsymbol{\theta}_h), 
\label{mix_dep}
\end{equation}
where $\pi_h({\bf x})=\nu_h({\bf x}) \prod_{l=1}^{h-1}\{1-\nu_l({\bf x})\}$ are predictor--dependent mixing probabilities having a stick-breaking representation \citep{SE94}, and $ K_{{\bf x}}(y ; \boldsymbol{\theta}) $ is a predictor--dependent kernel function indexed by $\boldsymbol{\theta}$. 

\begin{figure*}[t]
\includegraphics[trim=1cm 10cm 0cm 10cm,clip,height=7.5cm, width=20cm]{graphical}
\caption{Representation of the sequential mechanism to sample $G_i$.}\label{F_decision}
\end{figure*}

To provide a constructive representation of the   \textsc{lsbp}, let us first consider an equivalent formulation of the predictor--dependent mixture model in \eqref{mix_dep}. In particular, following standard hierarchical representations of mixture models, independent samples $y_1, \ldots, y_n$ from the random variable with density function displayed in \eqref{mix_dep}, can be obtained from
\vspace{4pt}
\begin{eqnarray}
\begin{split}
(y_i \mid  G_i= h, {\bf x}_i )&\sim K_{{\bf x}_i}(y_i ; \boldsymbol{\theta}_h), \\
  \mbox{pr}(G_i=h \mid {\bf x}_i)&=\pi_h({\bf x}_i)=\nu_h({\bf x}_i) \prod_{l=1}^{h-1}\{1-\nu_l({\bf x}_i)\}, \ \ \
 \end{split}
\label{marg}
\end{eqnarray}
for every unit $i=1, \ldots, n$, where $\boldsymbol{\theta}_h \sim P_0$ and $G_i \in \{1, 2, \ldots, +\infty\}$ the categorical variable denoting the mixture component associated with unit $i$. According to \eqref{marg}, every $G_i$ has probability mass function  $\mbox{p}(G_i \mid {\bf x}_i)=\prod_{h=1}^{+\infty}\pi_{h}({\bf x}_i)^{\mathbbm{1}(G_i=h)}$,  with $\pi_h({\bf x}_i)=\nu_h({\bf x}_i) \prod_{l=1}^{h-1}\{1-\nu_l({\bf x}_i)\}$ and $\mathbbm{1}(\cdot)$ denoting the indicator function. Thus, re--writing  $\nu_1({\bf x}_i), \ldots, \nu_h({\bf x}_i), \ldots$ as a function of the mixing probabilities $\pi_1({\bf x}_i), \ldots, \pi_h({\bf x}_i), \ldots$ via 
\vspace{2pt}
\begin{equation}
 \nu_h({\bf x}_i)=\frac{\pi_h({\bf x}_i)}{1-\sum_{l=1}^{h-1}\pi_l({\bf x}_i)}=\frac{\mbox{pr}(G_i=h \mid {\bf x}_i)}{\mbox{pr}(G_i>h-1 \mid {\bf x}_i)}, 
 \label{cont_ratio}
\end{equation}

\vspace{-1pt}

\noindent for each $h=1, \ldots, +\infty$, allows each  $\nu_h({\bf x}_i)$ to be easily interpreted as the probability of being allocated to component $h$, conditionally on the event of surviving to the previous $1, \ldots, h-1$ components---i.e. $\nu_h({\bf x}_i)=\mbox{pr}(G_i=h \mid G_i>h-1, {\bf x}_i)$. This result provides a formal characterization of the stick-breaking construction in \eqref{marg} as the continuation--ratio parameterization \citep{TU91} of the probability mass function for each component membership variable $G_i$, $i=1, \ldots, n$. This relevant connection with the literature on sequential inference for categorical data is common to all the stick-breaking priors, and provides substantial benefits for interpretation and inference. However, to our knowledge, this characterization---although  implicit in some computational routines \citep[e.g.][]{DP08,RDC11,RD11}, and recently adopted in Bayesian multinomial regression \citep{LIND2015}---has not yet been explicitly discussed and  fully exploited to facilitate interpretation and computation in Bayesian density regression---when $G_i$, $i=1, \ldots, n$ are latent variables. Indeed, as we describe in Section~\ref{Section3}, this characterization facilitates the implementation of  different routine--use algorithms in Bayesian inference, and provides a simple generative process for each $G_i$ which motivates the logistic link. 


In particular, according to Figure \ref{F_decision}, in the first step of this continuation--ratio generative mechanism, unit $i$ is either assigned to the first component  with probability $\nu_1({\bf{x}}_i)$ or to one of the others with complement probability. If $G_i=1$ the process stops, otherwise  it continues considering the reduced space $\{2, \ldots,+\infty\}$. A generic step $h$ is reached if $i$ has not been assigned to $1, \ldots, h-1$, and the decision at this step will be to either allocate $i$ to component $h$ with probability $\nu_h({\bf{x}}_i)$  or to one of the subsequent components $h+1, \ldots, +\infty$ with probability $1-\nu_h({\bf{x}}_i)$, conditioned on $G_i \in\{h, \ldots, +\infty\}$. Based on this representation, each indicator $\mathbbm{1}(G_i=h)=\zeta_{ih}$, $i=1, \ldots, n$---denoting the assignment to component $h$---can be expressed as
\begin{equation}
\zeta_{ih}=z_{ih} \prod_{l=1}^{h-1}(1-z_{il}),  \quad  (z_{ih} \mid {\bf{x}}_i) \sim \mbox{Bern}\{\nu_h({\bf{x}}_i)\},
\label{bin_seq}
\end{equation}
for each $h=1, \ldots, +\infty$, where $z_{ih}$ is a Bernoulli variable denoting the decision at the $h$th step to either allocate $i$ to component $h$ or to one of the subsequents $h+1, \ldots, +\infty$. Hence, according to \eqref{bin_seq}, the  sampling of each $G_i$, under the predictor--dependent stick-breaking representation for each $\pi_h({\bf{x}}_i)$ in \eqref{marg}, can be reformulated as a set of sequential Bernoulli choices  with natural parameters $\eta_h({\bf x}_i )=\mbox{logit}\{\nu_h({\bf{x}}_i )\}=\log[\nu_h({\bf x}_i) /\{1-\nu_h({\bf x}_i)\}]$, $h=1, \ldots, +\infty$, and logistic canonical link, under an exponential family representation. This result motivates the logit stick-breaking factorization
\begin{equation}
\pi_h({\bf x}_i)=\frac{\exp\{\eta_h({\bf x}_i )\}}{1+\exp\{\eta_h({\bf x}_i )\}} \prod_{l=1}^{h-1}\left[\frac{1}{1+\exp\{\eta_l({\bf x}_i )\}}\right],   \label{logit_stick}
\end{equation}
for each $h=1, \ldots, +\infty$, while allowing each $\eta_h({\bf x}_i )$ to be explicitly interpreted as the log-odds of the probability of being allocated to component $h$ or to one of the subsequents $h+1, \ldots, +\infty$, conditionally on the event of surviving to the first $1, \ldots, h-1$ components. This result facilitates prior specification and posterior inference for the stick-breaking weights, while allowing recent computational advances in Bayesian logistic regression  \citep{PS13} to be inherited in our density regression problem. 


To conclude our Bayesian representation, we require priors for the log-odds $\eta_h({\bf{x}}_i)$ of every  $\nu_h({\bf{x}}_i)$, in the continuation--ratio logistic regressions. A natural choice---consistent with classical generalized linear models \citep[e.g.][]{NW72}---is to define $\eta_h({\bf{x}}_i)$ as a linear combination of selected functions of the covariates $\boldsymbol{\psi}({\bf{x}}_i)=\{\psi_1({\bf{x}}_i), \ldots, \psi_R({\bf{x}}_i)\}^{\intercal}$ and consider Gaussian priors for the coefficients, obtaining
\begin{eqnarray}
\eta_h({\bf{x}}_i)=\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h, \quad \mbox{with }\boldsymbol{\alpha}_h \sim \mbox{N}_{R}(\boldsymbol{\mu}_{\alpha}, \boldsymbol{\Sigma}_{\alpha}),
\label{pred}
\end{eqnarray}
for every $h=1, \ldots, +\infty$. Although the linearity assumption in \eqref{pred} may seem restrictive, note that flexible formulations for $\eta_h({\bf{x}}_i)$, including regression via splines and Gaussian processes, induce linear relations  in the coefficients. Moreover, as we will outline in Section~\ref{Section3}, the linearity assumption simplifies computations, while inducing a logistic-normal prior for each $\nu_h({\bf{x}}_i)$, with well defined moments \citep{AS80}. 


Based on the above discussion, the logit stick-breaking does not induce Beta distributed stick-breaking weights, and therefore cannot be included in the general class of stick-breaking priors discussed in  \cite{IJ01}. However---as discussed in Section \ref{Sub2}---adapting the theoretical results in \cite{DP08} and \cite{RD11} to our logistic link, it can be shown that many relevant properties characterizing the priors discussed  in  \cite{IJ01} are met also under our case. Hence, the  \textsc{lsbp} is highly similar in its probabilistic nature and properties to other popular predictor--dependent stick-breaking constructions  \citep[e.g.][]{DP08, RD11}. However, as discussed in Section \ref{Section3}, differently from the current representations, the \textsc{lsbp} facilitates tractable computations under a broad variety of algorithms for posterior inference. %crucially improves computational tractability and enlarges the set of possible algorithms under which Bayesian inference can be carried out. 

\subsection{Properties of the logit stick-breaking prior}
\label{Sub2}
Let $\boldsymbol{\Theta}$ be a complete and separable metric space endowed with the Borel $\sigma$-algebra $\mathcal{B}(\boldsymbol{\Theta})$, and let $\{P_{\bf{x} }: {\bf{x}} \in \mathcal{X}\}$  denote the class of predictor--dependent random probability measures on  $\boldsymbol{\Theta}$, induced by the  \textsc{lsbp} 
\begin{eqnarray}
P_{\bf{x} }(\cdot) &=& \sum_{h=1}^{+\infty} \pi_h({\bf{x}} ) \delta_{\boldsymbol{\theta}_h}(\cdot), \label{stick} \\ \pi_h({\bf{x}} )  &=& \frac{\exp\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h\}}{1+\exp\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h\}} \prod_{l=1}^{h-1}\left[\frac{1}{1+\exp\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_l\}}\right] \nonumber
\end{eqnarray}
with  independent atoms $\boldsymbol{\theta}_h \sim P_0$, $h=1, \ldots, +\infty$ from the space $\{\boldsymbol{\Theta}, \mathcal{B}(\boldsymbol{\Theta}) \}$, and $\boldsymbol{\alpha}_h \sim \mbox{N}_{R}(\boldsymbol{\mu}_{\alpha}, \boldsymbol{\Sigma}_{\alpha})$ independently for every $h=1, \ldots, +\infty$. As discussed in Section~\ref{Section2}, representation~\eqref{stick} does not provide Beta distributed priors for the logit stick-breaking weights $\nu_h({\bf x})$, $h=1,\dots,+\infty$. However, in line with the random measure outlined in \cite{IJ01}, also the \textsc{lsbp} provides a well defined predictor--dependent random probability measure $P_{\bf{x} }$ at every $ {\bf{x}} \in \mathcal{X}$. This property is formalized  in Proposition~\ref{prop1}.

\begin{proposition} For any fixed $ {\bf{x}} \in \mathcal{X}$, $\sum_{h=1}^{+\infty} \pi_h({\bf{x}}) = 1$ almost surely, with $\pi_h({\bf{x}})$ factorized as in \eqref{stick} and $\boldsymbol{\alpha}_h \sim \mbox{\normalfont N}_{R}(\boldsymbol{\mu}_{\alpha}, \boldsymbol{\Sigma}_{\alpha})$ independently for every $h=1, \ldots, +\infty$.
 \label{prop1}
\end{proposition} 
\vspace{-5pt}
\noindent \textbf{Proof:} {\normalfont Recalling results in \cite{IJ01}, $\sum_{h=1}^{+\infty} \pi_h({\bf{x}}) = 1$ almost surely if and only if the equality $\sum_{h=1}^{+\infty}\mbox{E} [\log\{1-\nu_h({\bf{x}}) \}] = -\infty$ holds. Since $\log\{1-\nu_h({\bf{x}}) \}$ is concave in $\nu_h({\bf{x}})$ for every ${\bf{x}} \in \mathcal{X}$ and $h=1, \ldots, +\infty$, by the Jensen inequality  $\mbox{E} [\log\{1-\nu_h({\bf{x}}) \}] \leq \log[1-\mbox{E}\{ \nu_h({\bf{x}})\} ]$. Therefore, since $\nu_h({\bf{x}}) \in (0,1)$, from the usual properties of the expectation we have that $0<\mbox{E}\{ \nu_h({\bf{x}})\}=\mu_{1\nu}({\bf{x}})<1$, thereby providing $\log\{1-\mu_{1\nu}({\bf{x}})\} <0$. Leveraging these results, the proof of Proposition \ref{prop1} follows after noticing that  $\sum_{h=1}^{+\infty}\mbox{E} [\log\{1-\nu_h({\bf{x}}) \}] \leq \sum_{h=1}^{+\infty}\log\{1-\mu_{1\nu}({\bf{x}})\}= -\infty$. \qed}

\vspace{10pt}

Such property holds also for truncated models based on a finite number of components $H$. Indeed, consistent with  \cite{IJ01}, in this case it suffices to model the first $H-1$ weights $\nu_1({\bf{x}}), \ldots, \nu_{H-1}({\bf{x}})$ and let $\nu_H({\bf{x}})=1$ for any $ {\bf{x}} \in \mathcal{X}$, to ensure $\sum_{h=1}^{H} \pi_h({\bf{x}}) = 1$.


Results in Proposition~\ref{prop1} motivate further analyses of the logit stick-breaking prior. In particular, consistent with the theoretical studies on other stick-breaking priors not belonging to the class  discussed in \cite{IJ01}---e.g. \cite{DP08,RD11}---Proposition \ref{prop2} provides additional insights on the moments of the predictor--dependent random probability measure induced by our logit stick-breaking prior.


\begin{proposition} For every $ {\bf{x}} \in \mathcal{X}$ and $B \in \mathcal{B}(\boldsymbol{\Theta})$ the expectation of $P_{\bf{x} }(B) $ is $\mbox{\normalfont E}\{P_{\bf{x} }(B) \}=P_0(B)$, whereas the variance of $P_{\bf{x} }(B)$ for any truncated version of $P_{\bf{x} }(\cdot)$ in \eqref{stick}  with $H>1$---including the infinite case---is 
\begin{equation*}
\begin{split}
&\mbox{\normalfont var}\{P_{\bf{x} }(B) \}=P_0(B)\{1 - P_0(B)\}\times \\
 & \ \times \frac{\mu_{2\nu}({\bf x})\{ 1 - [1 - 2\mu_{1\nu}({\bf x}) + \mu_{2\nu}({\bf x})]^H\} }{2\mu_{1\nu}({\bf x}) - \mu_{2\nu}({\bf x})},
\end{split}
\end{equation*}
where $\mu_{1\nu}({\bf{x}})=\mbox{\normalfont E}\{ \nu_h({\bf{x}})\}$ and $\mu_{2\nu}({\bf{x}})=\mbox{\normalfont E}\{ \nu_h({\bf{x}})^2\}$ for every $h=1, \ldots, +\infty$. The covariance at two different predictor values $\bf{x}\in \mathcal{X}$ and $\bf{x}'\in \mathcal{X}$, $\bf{x} \neq \bf{x}'$, is instead 
\begin{eqnarray*}
\begin{split}
&\mbox{\normalfont cov}\{ P_{\bf{x}}(B),P_{\bf{x}'}(B)\} = P_0(B)\{1 - P_0(B)\} \times  \\
&\ \times \frac{\mu_{2\nu}({\bf{x}},{\bf{x}}')\{ 1 - [1 - \mu_{1\nu}({\bf{x}}) - \mu_{1\nu}({\bf{x}}') + \mu_{2\nu}({\bf{x}},{\bf{x}}')]^H\} }{\mu_{1\nu}({\bf{x}}) + \mu_{1\nu}({\bf{x}}') - \mu_{2\nu}({\bf{x}},{\bf{x}}')}
\end{split}
\end{eqnarray*}
 with $\mu_{2\nu}({\bf{x}},{\bf{x}}')=\mbox{\normalfont E}\{ \nu_h({\bf{x}})\nu_h({\bf{x}}')\}$.
 \label{prop2}
\end{proposition} 
\vspace{-5pt}
\noindent \textbf{{Proof:}} Results are a direct consequence of the calculations in Appendix 2 and Appendix 6 in \cite{RD11}, after replacing the probit link with the logistic one. \qed

\vspace{10pt}

According to Proposition \ref{prop2}, the expectation of $P_{{\bf{x}}}(\cdot)$ coincides with the base measure $P_0(\cdot)$ which can be therefore interpreted as the prior guess for the mixing measure at any $ {\bf{x}} \in \mathcal{X}$. This quantity is predictor--independent, meaning that a priori we are not forcing particular dependence structure between the atoms $\boldsymbol{\theta}$ and the predictors. The variance changes instead with the predictors via a function of the first two moments of the logistic-normal stick-breaking weights. Note that, since each $\nu_h({\bf{x}})$ is bounded between $0$ and $1$, we have $\nu_h({\bf{x}})\geq\nu_h({\bf{x}})^2$ for every $h=1, \ldots, +\infty$ and $ {\bf{x}} \in \mathcal{X}$, implying $0< \mu_{2\nu}({\bf{x}}) \leq \mu_{1\nu}({\bf{x}}) <1$. These results provide the bound $1 - 2\mu_{1\nu}({\bf x}) + \mu_{2\nu}({\bf x})<1$, which leads to a well defined limiting variance for the infinite case $H \rightarrow +\infty$ equal to $P_0(B)\{1 - P_0(B)\}  \mu_{2\nu}({\bf x})\{2\mu_{1\nu}({\bf x}) - \mu_{2\nu}({\bf x})\}^{-1}$. The limiting covariance is instead $P_0(B)\{1 - P_0(B)\} \mu_{2\nu}({\bf{x}},{\bf{x}}')\{\mu_{1\nu}({\bf{x}}) + \mu_{1\nu}({\bf{x}}') - \mu_{2\nu}({\bf{x}},{\bf{x}}')\}^{-1}$, after noticing that $\mu_{1\nu}({\bf{x}}) \geq  \mu_{2\nu}({\bf{x}},{\bf{x}}')$, $\mu_{1\nu}({\bf{x}}') \geq  \mu_{2\nu}({\bf{x}},{\bf{x}}')$ and $1 - \mu_{1\nu}({\bf{x}}) - \mu_{1\nu}({\bf{x}}') + \mu_{2\nu}({\bf{x}},{\bf{x}}')<1$. Hence the association is always positive and increases the closer ${\bf{x}}$ is to ${\bf{x}}'$. 

Although Proposition \ref{prop2} provides simple expressions for $\mbox{\normalfont E}\{P_{\bf{x} }(B) \}$, $\mbox{\normalfont var}\{P_{\bf{x} }(B) \}$ and $\mbox{\normalfont cov}\{ P_{\bf{x}}(B),P_{\bf{x}'}(B)\} $, the calculation of these quantities requires the moments of logistic-normal priors for the stick-breaking weights, induced by representation \eqref{pred}. Unfortunately these quantities are not available in explicit form  \citep[e.g.][]{AS80}. However, Proposition \ref{prop3} provides a simple procedure to accurately approximate the moments of logit stick-breaking weights leveraging a connection with the probit stick-breaking priors.

\begin{proposition} 
The logit stick-breaking prior described in representation \eqref{pred}, can be accurately approximated by a probit stick-breaking process $\nu_h({\bf{x}})\approx \Phi\{\boldsymbol{\psi}({\bf{x}})^\intercal\bar{\boldsymbol{\alpha}}_h\}$, with $\bar{\boldsymbol{\alpha}}_h={\boldsymbol{\alpha}}_h\sqrt{\pi/8} \sim \mbox{\normalfont N}_{R}\{\sqrt{\pi/8}\boldsymbol{\mu}_{\alpha}, (\pi/8)\boldsymbol{\Sigma}_{\alpha}\}$, for every ${\bf{x}} \in \mathcal{X}$ and $h=1, \ldots, +\infty$.
 \label{prop3}
\end{proposition} 
\vspace{-5pt}
\noindent \textbf{{Proof:}} {\normalfont Consistent with results in \cite{AM81}, the logistic link $\{1+\exp(-\boldsymbol{\psi}({\bf{x}})^\intercal{\boldsymbol{\alpha}}_h)\}^{-1}$ can be accurately approximated by $\Phi\{\boldsymbol{\psi}({\bf{x}})^\intercal{\boldsymbol{\alpha}}_h\sqrt{\pi/8} \}$. Therefore 
\begin{eqnarray*}
\nu_h({\bf{x}})=\{1+\exp(-\boldsymbol{\psi}({\bf{x}})^\intercal{\boldsymbol{\alpha}}_h)\}^{-1} &\approx& \Phi\{\boldsymbol{\psi}({\bf{x}})^\intercal{\boldsymbol{\alpha}}_h\sqrt{\pi/8} \}\\ 
&=&\Phi\{\boldsymbol{\psi}({\bf{x}})^\intercal\bar{\boldsymbol{\alpha}}_h\},
\end{eqnarray*}
with $\bar{\boldsymbol{\alpha}}_h \sim \sqrt{\pi/8}\mbox{\normalfont N}_{R}(\boldsymbol{\mu}_{\alpha},\boldsymbol{\Sigma}_{\alpha})$.\qed}


According to Proposition \ref{prop3}, the \textsc{lsbp} can be approximated by a  \textsc{psbp}, up to a simple transformation of the prior for the coefficients $\boldsymbol{\alpha}_h$. This result allows simple approximation for the moments of the logistic-normal priors on the stick-breaking weights by rescaling those provided in \cite{RD11} for the \textsc{psbp}. Moreover, a researcher considering a \textsc{psbp}, could perform  inference leveraging our algorithms, after rescaling the prior for each $\boldsymbol{\alpha}_h$ by $\sqrt{8/\pi}$.



\section{Bayesian computational methods}
\label{Section3}

Although the  \textsc{lsbp} and the associated computational procedures apply to a wider set of dependent mixture models and kernels, we will mainly focus---for the sake of clarity---on the general class of predictor--dependent infinite mixtures of Gaussians

\vspace{-10pt}
\begin{equation}
\begin{split}
f_{{\bf x}}(y)&=& \int\frac{1}{\sigma} \phi\left\{\frac{y -\boldsymbol{\lambda}({\bf x})^\intercal \boldsymbol{\beta}}{\sigma} \right\} \mbox{d} P_{\bf{x}}(  \boldsymbol{\beta},\sigma), \\
&=&\sum_{h=1}^{+\infty} \pi_h({\bf x}) \frac{1}{\sigma_h} \phi\left\{\frac{y -\boldsymbol{\lambda}({\bf x})^\intercal  \boldsymbol{\beta}_h}{\sigma_h} \right\}, 
\end{split}
\label{mix_dep_gaus}
\end{equation}
with $\boldsymbol{\beta}_h=(\beta_{1h},\ldots,\beta_{Ph})^\intercal$ a vector of coefficients linearly related to selected functions of the observed predictors $\{\lambda_1({\bf x}), \ldots, \lambda_P({\bf x})\}^{\intercal}$, comprising the vector $\boldsymbol{\lambda}({\bf x})$. Formulation \eqref{mix_dep_gaus} provides a flexible construction \citep{BAR2012,PA013}, and is arguably the most  used in Bayesian density regression. Generalizations to other kernels will be also discussed. 

\begin{remark}
Although we focus on density regression via \eqref{mix_dep_gaus}, leveraging the prior for the mixing weights induced by \eqref{logit_stick}--\eqref{pred}, we emphasize that the properties discussed in Section \ref{Section2}, and the algorithms derived in Section  \ref{Section3}, hold under more general priors for the predictor--dependent log-odds of the stick-breaking weights. A relevant one can be obtained by replacing \eqref{pred} with a Gaussian process prior $\eta_h(\cdot) \sim \textsc{gp}(0,c_h)$ having squared exponential covariance function $c_h$, for $h=1, \ldots, +\infty$ \citep[e.g.][]{RW06}. According to \cite{PA013}, this assumption---combined with model \eqref{mix_dep_gaus}---guarantees full support and posterior consistency as long as the stick-breaking weights are obtained by a monotone differentiable mapping of the Gaussian process prior $\eta_h(\cdot) \sim \textsc{gp}(0,c_h)$. This is the case of the logistic mapping characterizing our \textsc{lsbp} construction.

Besides the above desirable properties, it shall be noticed that under the Gaussian process prior and letting ${\bf{x}}_1^*, \ldots, {\bf{x}}_R^*$ the set of unique values of ${\bf{x}}_1, \ldots, {\bf{x}}_n$, each $\eta_h({\bf{x}}_i)$ can be easily rewritten as in \eqref{pred}, with $\boldsymbol{\alpha}_h \sim \mbox{N}_{R}(\boldsymbol{0}, \boldsymbol{\Sigma}_{\alpha})$,  $\boldsymbol{\Sigma}_{\alpha[rr']}=c_h({\bf{x}}_r^*,{\bf{x}}_{r'}^*)$ and $\boldsymbol{\psi}({\bf{x}}_i)=\{\mathbbm{1}({\bf{x}}_i={\bf{x}}_1^*), \ldots, \mathbbm{1}({\bf{x}}_i={\bf{x}}_R^*) \}^\intercal$. This allows the properties discussed in Section \ref{Section2}, and the algorithms in Section  \ref{Section3} to be directly applied to the Gaussian process case.
\end{remark}

As mentioned in Section \ref{sec1}, we provide here  detailed derivation of three computational methods for Bayesian density regression under model \eqref{mix_dep_gaus}, with logit stick-breaking prior \eqref{pred} for the mixing weights. In particular we consider a Gibbs sampler converging to the exact posterior, an Expectation Maximization (\textsc{em}) algorithm for fast estimation, and  a global variational Bayes  (\textsc{vb}) approximation for scalable inference.  The algorithms associated with these methods are available at \url{https://github.com/tommasorigon/LSBP}, along with the code to reproduce the application in Section \ref{Section4}.

All the above computational methodologies exploit representation \eqref{marg} of model \eqref{mix_dep_gaus}---with   $K_{ {\bf x}_i}(y_i ; \boldsymbol{\theta}_h) =\mbox{N}(y_i;\boldsymbol{\lambda}({\bf x}_i)^\intercal \boldsymbol{\beta}_h, \sigma^2_h)$---and the continuation--ratio characterization of the logit stick-breaking prior in Section \ref{Section2}. In fact, conditioned on the component membership variables ${\bf{G}}=(G_1, \ldots, G_n)$, the model reduces to a set of  Gaussian linear regressions---one for every mixture component---allowing inference for the kernel parameters $\boldsymbol{\beta}_h$ and $ \sigma^{2}_h$, via standard methods when $\boldsymbol{\beta}_h \sim \mbox{N}_P(\boldsymbol{\mu}_{\beta},\boldsymbol{\Sigma}_{\beta})$ and $\sigma_h^{-2} \sim \mbox{Ga}(a_{\sigma},b_{\sigma})$,  $h=1, \ldots, +\infty$. Moreover, exploiting  ${\bf{G}}$, and the continuation--ratio representation,  inference for the stick-breaking parameters $\boldsymbol{\alpha}_h$, $h=1, \ldots, +\infty$ in \eqref{pred}, proceeds as in a Bayesian logistic regression for the data $(z_{ih} \mid {\bf{x}}_i) \sim \mbox{Bern}[\nu_h({\bf x}_i )=\{1+\exp(-\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h)\}^{-1}]$  in \eqref{bin_seq}, for each $i=1, \ldots, n$ and $h=1, \ldots, +\infty$. Adapting results from the recent P\'olya-Gamma data augmentation scheme \citep{PS13} to our statistical model, the updating of $\boldsymbol{\alpha}_h$, $h=1, \ldots, +\infty$ can be easily accomplished exploiting the following result:
\begin{equation}
\begin{split}
\mbox{p}_{{\bf{x}}_i}(z_{ih})&= \frac{0.5 \exp\{(z_{ih}-0.5)\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h \}}{\mbox{cosh}\{0.5\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h \}}, \\
f_{{\bf{x}}_i}(\omega_{ih})&=\frac{\exp[-0.5\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h\}^2\omega_{ih} ]f(\omega_{ih})}{[\mbox{cosh}\{0.5\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h \}]^{-1}},
\end{split}
\label{polya}
\end{equation}
independently for every $i=1, \ldots, n$ and $h=1, \ldots, +\infty$, where $f_{{\bf{x}}_i}(\omega_{ih})$ and  $f(\omega_{ih})$ are the density functions of the P\'olya-Gamma random variables $\mbox{PG}\{1,\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h\}$, and $\mbox{PG}\{1,0\}$, respectively. Hence, based on \eqref{polya}, the contribution to the augmented likelihood for each pair $(z_{ih}, \omega_{ih})$ is proportional to a Gaussian kernel for transformed data $(z_{ih}-0.5)/\omega_{ih}$, given that $\mbox{p}_{{\bf{x}}_i}(z_{ih})f_{{\bf{x}}_i}(\omega_{ih})$ $\propto  \exp[(z_{ih}-0.5)\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h -0.5\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h\}^2\omega_{ih}]$. This allows posterior inference under a classical Bayesian linear regression. Refer to \cite{CH13} for further theoretical properties of the P\'olya-Gamma scheme.

Before providing a detailed derivation of the different algorithms available under the \textsc{lsbp} construction, we first study how a truncated version of the random probability measure $P_{\bf{x} }(\boldsymbol{\theta})$  approximates the infinite process. Although there  are some computational methods for the infinite representation,  these algorithms are not necessarily more tractable than those relying on a finite truncation, and still require approximations. In line with  \cite{RD11} and \cite{RDC11}, we develop detailed routines based on a finite representation, and discuss generalizations to the infinite case. This choice allows more direct comparisons and---based on Theorem \ref{teo4}---provides an accurate approximation of the infinite representation.
\begin{theorem}
\label{teo4}
For a sample ${\bf{y}}= (y_1,\dots,y_n)^\intercal$ with covariates ${\bf{X}} = \{{\bf{x}}_1, \ldots, {\bf{x}}_n\}^\intercal$, let 
\begin{equation*}
\begin{split}
& f^{(H)}({\bf{y}} \mid{\bf{X}})=f_{\bf{X}}^{(H)}({\bf{y}}) =\mbox{\normalfont E}_{P^{(H)}_{{\bf{x}}_i}} \left\{\prod_{i=1}^n f_{{\bf{x}}_i}^{(H)}(y_i) \right\},\\
& \quad = \mbox{\normalfont E}_{P^{(H)}_{{\bf{x}}_i}}\left ( \prod_{i=1}^n \left[\int\frac{1}{\sigma} \phi\left\{\frac{y_i -\boldsymbol{\lambda}({\bf x}_i)^\intercal \boldsymbol{\beta}}{\sigma} \right\} \mbox{d} {P^{(H)}_{{\bf{x}}_i}}(  \boldsymbol{\beta},\sigma) \right ] \right),
\end{split}
\end{equation*}
the marginal joint density of the data based on a truncated version of the \textsc{lsbp}  in \eqref{logit_stick}--\eqref{pred} with $H$ components, and let $f_{\bf{X}}^{(\infty)}({\bf{y}})$ be the same quantity in the infinite case. Then $$||f_{\bf{X}}^{(H)}({\bf{y}})  - f_{\bf{X}}^{(\infty)}({\bf{y}})||_1  \leq 4\sum_{i=1}^n\{1-\mu_{1\nu}({\bf x}_i) \}^{H-1}.$$
\end{theorem}

\noindent \textbf{{Proof:}} {\normalfont Adapting the proof of Theorem 1 in \cite{IJ02} to our representation we have that
\begin{eqnarray*}
\begin{split}
||f_{\bf{X}}^{(H)}({\bf{y}})  - f_{\bf{X}}^{(\infty)}({\bf{y}})||_1 &\le 4 \left[ 1 - \mbox{E}\left\{\prod_{i=1}^n \sum_{h=1}^{H-1}\pi_h({\bf x}_i)\right\}\right]\\
&=4 \mbox{E}\left[ 1 -\prod_{i=1}^n \sum_{h=1}^{H-1}\pi_h({\bf x}_i)\right].
\end{split}
\end{eqnarray*}
Since $\sum_{h=1}^{H-1}\pi_h({\bf x}_i)\leq 1$, and $1=\prod_{i=1}^n1$, following Lemma 1 in page 358 of \citet{BI94}, we can write $ 1 -\prod_{i=1}^n \sum_{h=1}^{H-1}\pi_h({\bf x}_i)=\prod_{i=1}^n1 -\prod_{i=1}^n \sum_{h=1}^{H-1}\pi_h({\bf x}_i) \leq \sum_{i=1}^n\{1- \sum_{h=1}^{H-1}\pi_h({\bf x}_i) \}$.  This result provides
\begin{eqnarray*}
\begin{split}
||f_{\bf{X}}^{(H)}({\bf{y}})  - f_{\bf{X}}^{(\infty)}({\bf{y}})||_1 &\leq 4[n-\sum_{i=1}^n \sum_{h=1}^{H-1}\mbox{E}\{\pi_h({\bf x}_i)\}],\\
\end{split}
\end{eqnarray*}
with $\sum_{h=1}^{H-1}\mbox{E}\{\pi_h({\bf x}_i)\}=\sum_{h=1}^{H-1}\mu_{1\nu}({\bf x}_i) \{1-\mu_{1\nu}({\bf x}_i) \}^{H-1}$ $=  1-\{1-\mu_{1\nu}({\bf x}_i) \}^{H-1}$. Substituting this quantity in $4[n-\sum_{i=1}^n \sum_{h=1}^{H-1}\mbox{E}\{\pi_h({\bf x}_i)\}]$, we obtain the final bound $4\sum_{i=1}^n\{1-\mu_{1\nu}({\bf x}_i) \}^{H-1}$.
\qed}

\vspace{10pt}

\begin{algorithm*}[t!]
 \caption{Steps of the Gibbs sampler for predictor--dependent finite mixtures of Gaussians} 
 \label{algo1}
     \Begin{\vspace{3pt}
 {\bf [1]}  Assign each unit $i=1, \ldots, n$ to a mixture component $h=1, \ldots, H$\;
 \For(){$i$ \mbox{from} $1$ to $n$}
 {
    Sample  $G_i \in (1, \ldots, H)$ from the categorical variable with probabilities $$\mbox{pr}(G_i = h \mid -) = \frac{\left [\nu_h({\bf{x}}_i)\prod_{l=1}^{h-1}\{1 - \nu_l({\bf{x}}_i)\} \right ]\frac{1}{\sigma_h} \phi\left\{\frac{y_i -\boldsymbol{\lambda}({\bf{x}}_i)^\intercal  \boldsymbol{\beta}_h}{\sigma_h} \right\}}{\sum_{q=1}^H\left [\nu_q({\bf{x}}_i)\prod_{l=1}^{q-1}\{1 - \nu_l({\bf{x}}_i)\} \right ]\frac{1}{\sigma_q} \phi\left\{\frac{y_i -\boldsymbol{\lambda}({\bf{x}}_i)^\intercal  \boldsymbol{\beta}_q}{\sigma_q} \right\}}, $$
for every $h=1, \ldots, H$.      }
\vspace{3pt}
 {\bf [2]}  Update the parameters ${\boldsymbol{\alpha}}_h$, $h=1, \ldots, H-1$, for the \textsc{lsbp}, exploiting the continuation--ratio representation and the results from the P\'olya-Gamma data augmentation in \eqref{polya}\;
 \For(){$h$ \mbox{from} $1$ to $H-1$}
 { \For(){every $i$ \mbox{such that} $G_i>h-1$}
 {Sample the P\'olya-Gamma data $\omega_{ih}$ from $(\omega_{ih} \mid -) \sim \mbox{PG}\{1,\boldsymbol{\psi}({\bf{x}}_i)^\intercal{\boldsymbol{\alpha}}_h\}.$} 
 Given  the P\'olya-Gamma data, update ${\boldsymbol{\alpha}}_h$ from the full conditional $({\boldsymbol{\alpha}}_h \mid -) \sim \mbox{N}_R(\boldsymbol{\mu}_{\boldsymbol{\alpha}_h},\boldsymbol{\Sigma}_{\boldsymbol{\alpha}_h}),$ having $\boldsymbol{\mu}_{\boldsymbol{\alpha}_h}=\boldsymbol{\Sigma}_{\boldsymbol{\alpha}_h}\{\boldsymbol{\Psi}_h({\bf {x}})^\intercal \boldsymbol{\kappa}_h+ \boldsymbol{\Sigma}_{\boldsymbol{\alpha}}^{-1}\boldsymbol{\mu}_{\boldsymbol{\alpha}}\}$,  $\boldsymbol{\Sigma}_{\boldsymbol{\alpha}_h}=\{\boldsymbol{\Psi}_h({\bf {x}})^\intercal \boldsymbol{\Omega}_h\boldsymbol{\Psi}_h({\bf {x}})+\boldsymbol{\Sigma}_{\boldsymbol{\alpha}}^{-1}\}^{-1}$, $\boldsymbol{\Omega}_h=\mbox{diag}(\omega_{i1}, \ldots, \omega_{i\bar{n}_h})$ and $\boldsymbol{\kappa}_h=(z_{i1}-0.5, \ldots, z_{i\bar{n}_h}-0.5)^\intercal$, with $z_{ih}=1$ if $G_i=h$ and $z_{ih}=0$ if $G_i>h$. }
\vspace{3pt}
{\bf [3]} Update the kernel parameters ${\boldsymbol{\beta}}_h$, $h=1, \ldots, H$, in \eqref{mix_dep_gaus}, leveraging results from standard Bayesian linear regression\;
 \For(){$h$ \mbox{from} $1$ to $H$}
 {
    Sample the coefficients comprising  ${\boldsymbol{\beta}}_h$ from the full conditional   $({\boldsymbol{\beta}}_h \mid -) \sim \mbox{N}_P(\boldsymbol{\mu}_{\boldsymbol{\beta}_h},\boldsymbol{\Sigma}_{\boldsymbol{\beta}_h})$, with $\boldsymbol{\mu}_{\boldsymbol{\beta}_h}=\boldsymbol{\Sigma}_{\boldsymbol{\beta}_h}\{\boldsymbol{\Lambda}_h({\bf {x}})^\intercal \boldsymbol{\Gamma}_h {\bf{y}}_h+ \boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\boldsymbol{\mu}_{\boldsymbol{\beta}}\}$,  $\boldsymbol{\Sigma}_{\boldsymbol{\beta}_h}=\{\boldsymbol{\Lambda}_h({\bf {x}})^\intercal \boldsymbol{\Gamma}_h\boldsymbol{\Lambda}_h({\bf {x}})+\boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\}^{-1}$, $\boldsymbol{\Gamma}_h= \sigma^{-2}_h \mbox{I}_{n_h \times n_h}$ and ${\bf{y}}_h$ the $n_h \times 1$ vector containing the responses for all the units with $G_i=h$.}
\vspace{3pt}
{\bf [4]} Update the precision parameters $\sigma^{-2}_h$, $h=1, \ldots, H$ of each kernel in \eqref{mix_dep_gaus}\;
 \For(){$h$ \mbox{from} $1$ to $H$}
 {
Sample $\sigma^{-2}_h$ from $(\sigma^{-2}_h \mid -) \sim \mbox{Ga}[a_\sigma +  0.5\sum_{i=1}^n\mathbbm{1}(G_i=h), b_\sigma + 0.5\sum_{i : G_i=h}\{y_i - \boldsymbol{\lambda}({\bf x}_i)^\intercal  \boldsymbol{\beta}_h\}^2]$.}
    }
\end{algorithm*}

According to Theorem  \ref{teo4}, for fixed $n$ and ${\bf{X}}$, the total variation distance between $f_{\bf{X}}^{(H)}({\bf{y}})$ and $f_{\bf{X}}^{(\infty)}({\bf{y}})$ vanishes as $H \rightarrow +\infty$, meaning that $f_{\bf{X}}^{(H)}({\bf{y}})$ converges in distribution to $f_{\bf{X}}^{(\infty)}({\bf{y}})$ when $H \rightarrow +\infty$. This rate of decay is exponential in $H$, and therefore the number of components has not to be very large in practice to accurately approximate the infinite representation.




\subsection{MCMC via Gibbs sampling}\label{gibbs}

\vspace{-6pt}
In deriving the Gibbs sampling algorithm for the model in  \eqref{mix_dep_gaus}, with \textsc{lsbp} in \eqref{logit_stick}--\eqref{pred}, we focus on a dependent mixture of Gaussians with fixed $H$. Although we do not derive it in detail here, the generalization to the infinite case is also possible, and can be easily incorporated leveraging the slice samplers of \cite{WA07} and \cite{KGW11}, which introduce a set of augmented latent variables allowing each step of the Gibbs sampler to rely on a finite representation. Such strategy slices the infinite mixture model, reducing it to a finite dimensional problem with $\bar{H}$ mixture components, where $\bar{H}$ varies stochastically at each step.  

Let $\boldsymbol{\Lambda}_h({\bf {x}})$ and $\boldsymbol{\Psi}_h({\bf {x}})$ denote the $n_h \times P$ and the $\bar{n}_h \times R$ predictor matrices in  \eqref{mix_dep_gaus} and  \eqref{pred} having row entries $\boldsymbol{\lambda}({\bf x}_i)^\intercal$ and $\boldsymbol{\psi}({\bf x}_i)^\intercal$, for only those statistical units $i$ such that $G_i =h$ and $G_i > h-1$, respectively, the Gibbs sampler for the truncated representation of model   \eqref{mix_dep_gaus} alternates between the full conjugate updating steps in Algorithm \ref{algo1}. Step {\bf [1]} can be run in parallel across units $i=1, \ldots, n$, whereas parallel computing for the different mixture components $h=1, \ldots, H$ can be easily implemented in steps  {\bf [2]}, {\bf [3]} and {\bf [4]}. 


\begin{algorithm*}[t!]
 \caption{Steps of the \textsc{em} algorithm  for predictor--dependent finite mixtures of Gaussians} 
 \label{algo2}
 
     \Begin{\vspace{3pt}
      Let $  \boldsymbol{\gamma}^{(t)}=( \boldsymbol{\alpha}^{(t)}$, $\boldsymbol{\beta}^{(t)}$, $\boldsymbol{\sigma}^{2(t)})$ denote the values of the parameters at iteration $t$.
      
      \vspace{5pt}
 {\bf [1] Expectation:}   Exploiting results in \eqref{EM}, the expectation of \eqref{eq::completeloglik} with respect to the augmented data $(\boldsymbol{\zeta}_i, \bar{\boldsymbol{\omega}}_i)$, for each $i=1, \ldots, n$, can be simply obtained by plugging in $\hat{\boldsymbol{\zeta}}_i=\mbox{E}(\boldsymbol{\zeta}_i\mid {{y}}_i, {\bf{x}}_i, \boldsymbol{\beta}^{(t)}, \boldsymbol{\sigma}^{2(t)})$ and $\hat{\bar{\boldsymbol{\omega}}}_i=\mbox{E}(\bar{{\boldsymbol{\omega}}}_i\mid  {\bf{x}}_i,\hat{\boldsymbol{\zeta}}_i, \boldsymbol{\alpha}^{(t)})$ in \eqref{EM}. Therefore:
 
 \For(){$i$ \mbox{from} $1$ to $n$}
 {\For(){$h$ \mbox{from} $1$ to $H$}
 {Compute $\hat{{{\zeta}}}_{ih}$ by applying the following expression $$\hat{{{\zeta}}}_{ih} = \frac{\left [\nu_h^{(t)}({\bf{x}}_i)\prod_{l=1}^{h-1}\{1 - \nu_l^{(t)}({\bf{x}}_i)\} \right ]\frac{1}{\sigma_h^{(t)}} \phi\left\{\frac{y_i -\boldsymbol{\lambda}({\bf{x}}_i)^\intercal  \boldsymbol{\beta}_h^{(t)}}{\sigma_h^{(t)}} \right\}}{\sum_{q=1}^H\left [\nu_q^{(t)}({\bf{x}}_i)\prod_{l=1}^{q-1}\{1 - \nu_l^{(t)}({\bf{x}}_i)\} \right ]\frac{1}{\sigma_q^{(t)}} \phi\left\{\frac{y_i -\boldsymbol{\lambda}({\bf{x}}_i)^\intercal  \boldsymbol{\beta}_q^{(t)}}{\sigma_q^{(t)}} \right\}},$$ and calculate $\hat{\bar{{\omega}}}_{ih}$ via $\hat{\bar{{\omega}}}_{ih} =\{2\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h^{(t)}\}^{-1}\tanh{\{0.5\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h^{(t)}\}}\sum_{l=h}^H\hat{\zeta}_{il}$, \citep{PS13}.} }
\vspace{3pt}
 {\bf [2] Maximization:}  To maximize the expected complete log-posterior $\mbox{log} f(\boldsymbol{\alpha}, \boldsymbol{\beta}, \boldsymbol{\sigma}^{2}\mid {\bf{y}},\hat{\boldsymbol{\zeta}},\hat{\bar{\boldsymbol{\omega}}}, {\bf{x}} )$, note that according to   \eqref{eq::completeloglik}--\eqref{EM}, modes $\boldsymbol{\alpha}^{(t+1)}$ and $(\boldsymbol{\beta}^{(t+1)},\boldsymbol{\sigma}^{2(t+1)} )$ can be obtained separately as follow:
 
 \For(){$h$ \mbox{from} $1$ to $H-1$}
 {To compute $\boldsymbol{\alpha}_h^{(t+1)}$, note that since $\boldsymbol{\alpha}_h$ has Gaussian prior, and provided that the second term in \eqref{EM} is based on Gaussian kernels, the estimated  $\boldsymbol{\alpha}_h$ at step $t+1$ coincides with the mean of a full conditional Gaussian, similar to the one in step {\bf{[2]}} of Algorithm \ref{algo1}.
 $$\boldsymbol{\alpha}_h^{(t+1)}=\{\boldsymbol{\Psi}({\bf {x}})^\intercal \mbox{diag}(\hat{\bar{\omega}}_{1h}, \ldots, \hat{\bar{\omega}}_{nh})\boldsymbol{\Psi}({\bf {x}})+\boldsymbol{\Sigma}_{\boldsymbol{\alpha}}^{-1}\}^{-1}\{\boldsymbol{\Psi}({\bf {x}})^\intercal (\hat{\bar{\kappa}}_{1h}, \ldots, \hat{\bar{\kappa}}_{nh})^\intercal+ \boldsymbol{\Sigma}_{\boldsymbol{\alpha}}^{-1}\boldsymbol{\mu}_{\boldsymbol{\alpha}}\} $$} 
  \For(){$h$ \mbox{from} $1$ to $H$}
 {A similar approach is can be considered to compute  $\boldsymbol{\beta}_h^{(t+1)}$ and $\boldsymbol{\sigma}_h^{2(t+1)}$ under the Gaussian and Inverse--Gamma priors for these parameters and the Gaussian kernel characterizing the first term in \eqref{EM}. Hence,  adapting steps {\bf{[3]}} and {\bf{[4]}} in Algorithm \ref{algo1} to the \textsc{em} setting, provides:
\begin{eqnarray*}
\boldsymbol{\beta}_h^{(t+1)}&{=}&\{\boldsymbol{\Lambda}({\bf {x}})^\intercal \hat{\boldsymbol{\Gamma}}^{(t)}_h\boldsymbol{\Lambda}({\bf {x}})+\boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\}^{-1}\{\boldsymbol{\Lambda}({\bf {x}})^\intercal\hat{\boldsymbol{\Gamma}}^{(t)}_h{\bf{y}}+ \boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\boldsymbol{\mu}_{\boldsymbol{\beta}}\},  \\
\boldsymbol{\sigma}_h^{-2(t+1)}&{=}&\mbox{max}\{0, [a_{\sigma}+0.5\sum_{i=1}^n \hat{\zeta}_{ih}-1][b_{\sigma}+0.5 \sum_{i=1}^n \hat{\zeta}_{ih}\{y_i-\boldsymbol{\lambda}({\bf{x}}_i)^\intercal \boldsymbol{\beta}_h^{(t)}\}^2]^{-1}\}
\end{eqnarray*}

\vspace{-10pt}
with $\hat{\boldsymbol{\Gamma}}^{(t)}_h=\sigma^{-2(t)}_{h} \mbox{diag}(\hat{\zeta}_{1h}, \ldots, \hat{\zeta}_{nh})$.} }
\end{algorithm*}



\subsection{EM algorithm} \label{em}
\vspace{-6pt}
In several situations, when either $P$  or $n$ are large, the Gibbs sampler described in Section \ref{gibbs} could face computational bottlenecks. If a point estimate of model \eqref{mix_dep_gaus} is the main quantity of interest---e.g. for prediction purposes---one possibility in these high-dimensional problems is to rely on a more efficient procedure specifically designed for this goal, such as the  \textsc{em} \citep{DL77}. The implementation of a simple \textsc{em} for a finite representation of model \eqref{mix_dep_gaus} with \textsc{lsbp} \eqref{logit_stick}--\eqref{pred}, greatly benefits from the P\'olya-Gamma data augmentation, which has analytical expectation and allows direct maximization within a Gaussian linear regression framework. Note that, although the \textsc{em} algorithm is commonly implemented for maximum likelihood estimation, it can be easily modified to estimate posterior modes  \citep[e.g.][]{DL77}. 


The  \textsc{em} proposed in Algorithm \ref{algo2} alternates between a maximization step for the parameters $(\boldsymbol{\alpha}_h, \boldsymbol{\beta}_h, \sigma_h^{2})$, $h=1, \ldots, H$, and an expectation step for the augmented data  $(\boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i)$, $i=1, \ldots, n$, with $\boldsymbol{\zeta}_{i}=\{\zeta_{i1}=\mathbbm{1}(G_i=1), \ldots, \zeta_{iH}=\mathbbm{1}(G_i=H) \}^\intercal$ the vector of binary indicators denoting membership to a mixture component, and  $\bar{\boldsymbol{\omega}}_i=(\bar{\omega}_{i1}, \ldots, \bar{\omega}_{iH})^{\intercal}$ the corresponding P\'olya-Gamma augmented data. Note that in this case we  work directly with the component indicator variables $\boldsymbol{\zeta}_{i}$ instead of the binary vectors ${\bf{z}}_i=(z_{i1}, \ldots, z_{iH})^{\intercal}$ in \eqref{bin_seq}, to facilitate simpler derivations. Indeed, also under this parameterization, the P\'olya-Gamma data augmentation can be easily considered, provided that---according to \eqref{marg}---$\mbox{p}(G_i \mid {\bf x}_i)$ can be expressed as
\begin{eqnarray}
\begin{split}
\mbox{p}(G_i \mid {\bf x}_i)&=\prod_{h=1}^{H}\pi_h( {\bf x}_i)^{\mathbbm{1}(G_i=h)}\\
&=\prod_{h=1}^{H} \nu_h( {\bf x}_i)^{\zeta_{ih}}\{ 1-\nu_h( {\bf x}_i)\}^{\sum_{l=h}^{H}\zeta_{il}-\zeta_{ih}},
\end{split}
\label{zeta}
\end{eqnarray}
for every unit $i=1, \ldots, n$. Based on the above quantities, the complete log-posterior $\mbox{log} f_{{\bf{x}}}(\boldsymbol{\alpha}, \boldsymbol{\beta}, \boldsymbol{\sigma}^{2}\mid {\bf{y}},\boldsymbol{\zeta},\bar{\boldsymbol{\omega}} )=\mbox{log} f_{{\bf{x}}}(\boldsymbol{\alpha}_1, \ldots, \boldsymbol{\alpha}_{H-1}, \boldsymbol{\beta}_1, \ldots, \boldsymbol{\beta}_H, \sigma_1^{2}, \ldots, \sigma_H^{2}\mid {\bf{y}},\boldsymbol{\zeta},\bar{\boldsymbol{\omega}} )$ underlying the proposed \textsc{em} routine, can be written---up to an additive constant---as
\begin{eqnarray}
\label{eq::completeloglik}
\begin{split}
&\sum_{i=1}^n\ell(\boldsymbol{\alpha}, \boldsymbol{\beta},  \boldsymbol{\sigma}^{2}; {y}_i,\boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i, {\bf{x}}_i ) +  \log{f(\boldsymbol{\alpha})f(\boldsymbol{\beta})f(\sigma^2)}
\end{split}
\end{eqnarray}
where $\ell(\boldsymbol{\alpha}, \boldsymbol{\beta},  \boldsymbol{\sigma}^{2}; {y}_i,\boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i, {\bf{x}}_i )$ is the contribution of unit $i$ to the complete log-likelihood, whereas $f(\boldsymbol{\alpha})$, $f(\boldsymbol{\beta})$, and $f({\sigma}^2)$ are the prior density functions for the model parameters. Working on $\ell(\boldsymbol{\alpha}, \boldsymbol{\beta},  \boldsymbol{\sigma}^{2}; {y}_i,\boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i, {\bf{x}}_i )$ has relevant benefits. Indeed, exploiting  equations \eqref{marg} and \eqref{zeta}, and the results in  \cite{PS13}, \cite{CH13} summarized in \eqref{polya}, $\ell(\boldsymbol{\alpha}, \boldsymbol{\beta},  \boldsymbol{\sigma}^{2}; {y}_i,\boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i, {\bf{x}}_i )$, can be factorized as
$\ell( \boldsymbol{\beta},  \boldsymbol{\sigma}^{2}; {y}_i,\boldsymbol{\zeta}_i, {\bf{x}}_i )+\ell{(\boldsymbol{\alpha}; \boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i, {\bf{x}}_i )},$
where the first summand equals to $$\sum_{h=1}^H\zeta_{ih}\left[-\frac{\{y_i -\boldsymbol{\lambda}({\bf x}_i)^\intercal  \boldsymbol{\beta}_h\}^2}{2\sigma_h^{2}} -\frac{1}{2}\log(\sigma_h^2)\right]+\mbox{const},$$ whereas the second coincides with $$\sum_{h=1}^{H-1}\bar{\kappa}_{ih}\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h+\sum_{h=1}^{H-1} \left[-\frac{\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h\}^2}{2\bar{\omega}_{ih}^{-1}}\right]+\mbox{const},$$ where $\bar{\kappa}_{ih}=\zeta_{ih}-0.5\sum_{l=h}^H\zeta_{ih}$. Hence, the contribution $\ell(\boldsymbol{\alpha}, \boldsymbol{\beta},  \boldsymbol{\sigma}^{2}; {y}_i,\boldsymbol{\zeta}_i,\bar{\boldsymbol{\omega}}_i, {\bf{x}}_i )$ of unit $i$ to the complete log-likelihood is


\vspace{-12pt}
\begin{eqnarray}
\begin{split}
&\sum_{h=1}^H\zeta_{ih}\left[-\frac{\{y_i -\boldsymbol{\lambda}({\bf x}_i)^\intercal  \boldsymbol{\beta}_h\}^2}{2\sigma_h^2} -\frac{1}{2}\log(\sigma_h^2)\right]+ \\
& \quad + \sum_{h=1}^{H-1}\left[\bar{\kappa}_{ih}\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h -\bar{\omega}_{ih}\frac{\{\boldsymbol{\psi}({\bf{x}}_i)^\intercal\boldsymbol{\alpha}_h\}^2}{2}\right]+ \mbox{const},
\end{split}
\label{EM}
\end{eqnarray}
where both terms in equation \eqref{EM} are linear in the augmented data $(\boldsymbol{\zeta}_i, \bar{\boldsymbol{\omega}}_i)$, and represent the sum of Gaussian kernels. This linearity property  simplifies computations in the expectation step for the complete log-posterior in equation \eqref{eq::completeloglik}, whereas the Gaussian structure allows simple maximizations.  Since the joint maximization of the expected complete log-posterior with respect to $(\boldsymbol{\beta},\boldsymbol{\sigma}^{2})$  is intractable, we rely on a conditional maximization procedure in the last step of Algorithm \ref{algo2}, which provides closed form solutions. This approach is referred as Expectation Conditional Maximization (\textsc{ecm}) \citep{MR93}. 


\subsection{Mean Field Variational Bayes}\label{vb}
Section \ref{em} provides a scalable procedure for estimation of posterior modes in large-scale problems. However, an appealing aspect of the Bayesian approach is in allowing uncertainty quantification via inference on the entire posterior distribution. As discussed in Section \ref{em}, the Gibbs sampler represents an appealing procedure which converges to the exact posterior, but faces computational bottlenecks. This motivates  scalable variational methods for approximate and tractable Bayesian inference \citep{BI06,Blei2017}.


We seek a variational distribution $q_{{\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega})$ that best approximates the joint posterior $f_{{\bf{x}}}( \boldsymbol{\alpha}, \boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf z }, $ $  \boldsymbol{\omega} \mid {\bf y})$, while maintaining simple computations. This can be obtained by minimizing the Kullback-Leibler divergence $\textsc{kl}\{q_{{\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega})\mid\mid f_{{\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega} \mid {\bf{y}})\}$ between the variational distribution and the full posterior, or alternatively by maximizing the evidence lower bound $\textsc{elbo}\{q_{{\bf{x}}}(\boldsymbol{\alpha}, \boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega} ) \}$ of the log-marginal density $\log f_{{\bf{X}}}^{(H)}({\bf{y}})$, provided that $\log f_{{\bf{X}}}^{(H)}({\bf{y}})$ can be analytically expressed  as the sum of the \textsc{elbo} and the positive \textsc{kl} divergence. This lower bound  to be maximized, can be expressed as
$$\mbox{E}_{\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega}} \left[\log{\left\{  \frac{ f_{\bf{x}}({\bf{y}}, \boldsymbol{\alpha}, \boldsymbol{\beta}, \boldsymbol{\sigma}^2,{\bf{z}}, \boldsymbol{\omega})}{q_{ {\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega})}\right\}} \right],$$
where the above expectation is taken with respect to the variational distribution $q_{{\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega})$.

\begin{algorithm*}[t!]
 \caption{Steps of the \textsc{vb} algorithm  for predictor--dependent finite mixtures of Gaussians} 
 \label{algo3}
\Begin{\vspace{3pt}
Let $q^{(t)}(\cdot)$ denote the generic variational distribution at iteration $t$.

\vspace{5pt}
{\bf [1]} Compute $q_{{\bf{x}}_i}^{*(t)}(z_{ih})$, for each $i=1, \ldots, n$ and $h=1, \ldots, H-1$;
 \For(){$i$ from $1$ to $n$}{
\For(){$h$  from $1$ to $H-1$}{
\vspace{5pt}
It can be easily shown that the optimal solution for the variational distribution of each $z_{ih}$ coincides with  $q_{{\bf{x}}_i}^{*(t)}(z_{ih})=\mbox{Bern}(\rho_{ih})$, with $$\mbox{logit}(\rho_{ih}) = \boldsymbol{\psi}({\bf{x}}_i)^\intercal\mbox{E}(\boldsymbol{\alpha}_h) + \sum_{l=h}^H \zeta_{il}^{(h)}\left[0.5\cdot\mbox{E}(\log{\sigma^{-2}_l}) -0.5\cdot\mbox{E}(\sigma^{-2}_l)\mbox{E}\{(y_i - \boldsymbol{\lambda}({\bf{x}}_i)^\intercal  \boldsymbol{\beta}_l)^2\}\right], $$
where the expectations are taken with the respect to the current variational distributions for the other parameters, whereas $\zeta^{(h)}_{il}=\prod_{r=1}^{l-1}(1-\rho_{ir})$ if $l=h$, and $\zeta^{(h)}_{il}=-\rho_{il}\prod_{r=1,r\neq h}^{l-1}(1-\rho_{ir})$ otherwise. Note also that $\rho_{iH}=1$.   }}

 {\bf [2]} Compute $q^{*(t)}_{{\bf{x}}}(\boldsymbol{\alpha}_{h})$, for each $h=1, \ldots, H-1$;

 
 \For(){$h$ \mbox{from} $1$ to $H-1$}{
  \vspace{5pt}
  
  The optimal solution $q^{*(t)}_{{\bf{x}}}(\boldsymbol{\alpha}_h)$ for the variational distribution of each $\boldsymbol{\alpha}_h$ is easily available as a Gaussian distribution $$q^{*(t)}_{{\bf{x}}}(\boldsymbol{\alpha}_h)=\mbox{N}_R[\{\boldsymbol{\Psi}({\bf{x}})^\intercal{\bf{V}}_h\boldsymbol{\Psi}({\bf{x}}) + \boldsymbol{\Sigma}_{\alpha}^{-1}\}^{-1}\{\boldsymbol{\Psi}({\bf{x}})^\intercal \boldsymbol{\rho}_h + \boldsymbol{\Sigma}_{\alpha}^{-1}\boldsymbol{\mu}_{\alpha}\},\{\boldsymbol{\Psi}({\bf{x}})^\intercal {\bf{V}}_h\boldsymbol{\Psi}({\bf{x}}) + \boldsymbol{\Sigma}_{\alpha}^{-1}\}^{-1}]$$ with ${\bf{V}}_h {=} \ \mbox{diag}\{\mbox{E}(\omega_{1h}),\dots,\mbox{E}(\omega_{nh})\}$ and $ \boldsymbol{\rho}_h{=}\ (\rho_{1h} - 0.5,\dots,\rho_{nh} - 0.5)$.}
  \vspace{3pt}
   {\bf [3]} Compute the variational distribution $q^{*(t)}_{{\bf{x}}_i}(\omega_{ih})$ for each $i=1, \ldots, n$ and $h=1, \ldots, H-1$ 
 
 \For(){$i$ from $1$ to $n$}{
\For(){$h$  from $1$ to $H-1$}{
\vspace{5pt}
Update the optimal solution $q^{*(t)}_{{\bf{x}}_i}(\omega_{ih})$, having P\'olya-gamma distribution with parameters
$$q^{*(t)}_{{\bf{x}}_i}(\omega_{ih})= \text{PG}\left(1, \xi_{ih}\right), \qquad \xi_{ih}^2 = \boldsymbol{\psi}({\bf{x}}_i)^\intercal \mbox{E}(\boldsymbol{\alpha}_h\boldsymbol{\alpha}_h^\intercal )\boldsymbol{\psi}({\bf{x}}_i).$$
From \citet{PS13}, recall that $\mbox{E}(\omega_{ih}) = 0.5\xi_{ih}^{-1}\tanh(0.5 \xi_{ih})$, with $\tanh(\cdot)$ denoting the hyperbolic tangent function. }}
  
  
\vspace{3pt}
 {\bf [4]} Compute $q^{*(t)}_{{\bf{x}}}(\boldsymbol{\beta}_{h})$ and  $q^{*(t)}_{{\bf{x}}}({\sigma}^2_{h})$ , for each $h=1, \ldots, H$;
 
  
  \For(){$h$ \mbox{from} $1$ to $H$}{
   \vspace{5pt}
   
Update the optimal solution for the variational distributions of $\boldsymbol{\beta}_h$ and $\sigma^{2}_h$ via
\begin{eqnarray*}
q^{*(t)}_{{\bf{x}}}(\boldsymbol{\beta}_h)= \mbox{N}_P[\{\boldsymbol{\Lambda}({\bf {x}})^\intercal {\boldsymbol{\Gamma}}_h\boldsymbol{\Lambda}({\bf {x}})+\boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\}^{-1}\{\boldsymbol{\Lambda}({\bf {x}})^\intercal{\boldsymbol{\Gamma}}_h{\bf{y}}+ \boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\boldsymbol{\mu}_{\boldsymbol{\beta}}\},\{\boldsymbol{\Lambda}({\bf {x}})^\intercal {\boldsymbol{\Gamma}}_h\boldsymbol{\Lambda}({\bf {x}})+\boldsymbol{\Sigma}_{\boldsymbol{\beta}}^{-1}\}^{-1}]&& \\
q^{*(t)}_{{\bf{x}}}(\boldsymbol{\sigma}^{-2}_h)=\mbox{Ga}[a_{\sigma}+0.5\sum_{i=1}^n \mbox{E}({\zeta}_{ih}), b_{\sigma}+0.5 \sum_{i=1}^n \mbox{E}({\zeta}_{ih})\mbox{E}\{y_i-\boldsymbol{\lambda}({\bf{x}}_i)^\intercal \boldsymbol{\beta}_h\}^2] \ \ \ \ \ \ \ \  \ \ \ \ \ \ \ \  \ \ \ \ \ \ \ \  \ &&
\end{eqnarray*}

\vspace{-10pt}
with ${\boldsymbol{\Gamma}}_h=\mbox{E}(\sigma^{-2}_{h}) \mbox{diag}\{\mbox{E}({\zeta}_{1h}), \ldots, \mbox{E}({\zeta}_{nh})\}$.}   
}
\end{algorithm*}

Without further restrictions, the Kullback-Leibler divergence is minimized when the variational distribution is equal to the true posterior distribution, which is analytically intractable. To address this issue, a common strategy in \textsc{vb} inference \citep[see e.g.][]{Blei2017} is to assume that the variational distribution belongs to a mean field variational family. This assumption forces a posteriori independence among distinct groups of parameters, implying that the variational distribution  can be expressed as the product of the marginals distributions. Specifically, we let the variational distribution to factorize in distinct groups 
\begin{equation}
\label{mean_field}
q_{\bf{x}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}},\boldsymbol{\omega}) = q_{\bf{x}}(\boldsymbol{\alpha}, \boldsymbol{\beta} )q_{\bf{x}}(\boldsymbol{\sigma}^2)\prod_{h=1}^{H-1}q_{\bf{x}}({\bf{z}}_h,\boldsymbol{\omega}_h),
\end{equation}
with ${\bf{z}}_h = (z_{1h},\dots,z_{nh})$ and $\boldsymbol{\omega}_h = (\omega_{1h},\dots,\omega_{nh})$. Note that we are not making any assumption about the functional form of the distributions $q_{\bf{x}}(\boldsymbol{\alpha}, \boldsymbol{\beta} )$, $q_{\bf{x}}(\boldsymbol{\sigma}^2 )$ and $\prod_{h=1}^{H-1}q_{\bf{x}}({\bf{z}}_h,\boldsymbol{\omega}_h)$, neither we are imposing any additional independence structure between the parameters. However, a closer look at the augmented likelihood function $f_{\bf{x}}({\bf{y}}, {\bf{z}}, \boldsymbol{\omega} \mid \boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2) = f_{\bf{x}}({\bf{y}} \mid  {\bf{z}}, \boldsymbol{\beta}, \boldsymbol{\sigma}^2)f_{\bf{x}}({\bf{z}} \mid  \boldsymbol{\alpha})f_{\bf{x}}(\boldsymbol{\omega} \mid \boldsymbol{\alpha}) $, defined as
\begin{eqnarray*}
\begin{split}
&\prod_{i=1}^n \prod_{h=1}^H \left(\frac{1}{\sigma_h}\phi\left\{\frac{y_i - \boldsymbol{\lambda}({\bf x}_i)^\intercal\boldsymbol{\beta}_h}{\sigma_h}\right\}\right)^{z_{ih}\prod_{l=1}^{h-1}(1-z_{il})} \times\\
&\quad  \times \prod_{i=1}^n \prod_{h=1}^{H-1} f(\omega_{ih}) \frac{\exp{\left\{ (z_{ih} - 0.5)\boldsymbol{\psi}({\bf x}_i)^\intercal\boldsymbol{\alpha}_h \right\}}}{\exp{\left\{ 0.5 \omega_{ih}(\boldsymbol{\psi}({\bf x}_i)^\intercal\boldsymbol{\alpha}_h )^2\right\}}},
\end{split}
\end{eqnarray*}
reveals that the variational distribution in equation~\eqref{mean_field} can be furtherly factorized exploiting the conditional independence structure between the parameters---which is apparent from the above equation---and independence among prior distributions. This additional simplification is sometimes called induced factorization \citep[Ch. 10.2.5]{BI06}, since it arises from the assumed mean field approximation~\eqref{mean_field} and the structure of the true posterior distribution. Combining~\eqref{mean_field} with the factorization induced by the full posterior $f_{{\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega} \mid {\bf{y}})$, we obtain
\begin{equation*}
\begin{split}
&q_{\bf{x}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega}) = \prod_{h=1}^{H-1}q_{\bf{x}}(\boldsymbol{\alpha}_h)\prod_{h=1}^{H}q_{\bf{x}}(\boldsymbol{\beta}_h)\prod_{h=1}^{H}q_{\bf{x}}({\sigma}^2_h) \times \\
&\qquad \qquad \qquad \qquad \quad \times \prod_{h=1}^{H-1}\prod_{i=1}^n q_{{\bf{x}}_i}(z_{ih})\prod_{h=1}^{H-1}\prod_{i=1}^n q_{{\bf{x}}_i}(\omega_{ih}).
\end{split}
\end{equation*}
Following \citet[Ch. 10]{BI06}, the optimal solutions for the variational distribution---under the factorization given in the equation above---have the following form
\vspace{-5pt}
\begin{equation*}
\begin{split}
&  \log{q_{{\bf{x}}}^{*}(\boldsymbol{\beta}_{h})}=\mbox{E}_{\boldsymbol{\sigma}^2,{\bf{z}}}[\log\{   f_{{\bf{x}}}({\bf{y}} \mid  {\bf{z}}, \boldsymbol{\beta}, \boldsymbol{\sigma}^2)f(\boldsymbol{\beta}_h)\}] + c_{\boldsymbol{\beta}_h},\\
&  \log{q_{{\bf{x}}}^{*}({\sigma}^2_{h})}=\mbox{E}_{\boldsymbol{\beta},{\bf{z}}}[\log\{   f_{{\bf{x}}}({\bf{y}} \mid  {\bf{z}}, \boldsymbol{\beta}, \boldsymbol{\sigma}^2)f({\sigma}^2_h)\}] + c_{{\sigma}^2_h}, 
  \end{split}
\end{equation*}
for every $h=1, \ldots, H$, and 
\begin{equation*}
\begin{split}
& \log{q_{{\bf{x}}_i}^{*}(z_{ih})}=\mbox{E}_{\boldsymbol{\alpha},\boldsymbol{\beta},\boldsymbol{\sigma}^2, {\bf{z}}_{i,-h}}[\log  f_{{\bf{x}}}({\bf{y}}, {\bf{z}} \mid \boldsymbol{\beta}, \boldsymbol{\sigma}^2  \boldsymbol{\alpha})] + c_{z_{ih}}, \\
&  \log{q_{{\bf{x}}_i}^{*}(\omega_{ih})}=\mbox{E}_{\boldsymbol{\alpha}}[\log f_{{\bf{x}}}(\omega_{ih} \mid \boldsymbol{\alpha})] + c_{\omega_{ih}}, \ i=1, \ldots, n, \\
& \log{q_{{\bf{x}}}^{*}(\boldsymbol{\alpha}_{h})}=\mbox{E}_{ {\bf{z}},\boldsymbol{\omega}}[\log\{  f_{{\bf{x}}}({\bf{z}},\boldsymbol{\omega} \mid  \boldsymbol{\alpha}) f(\boldsymbol{\alpha}_h)\}] + c_{\boldsymbol{\alpha}_h}, 
    \end{split}
\end{equation*}
for every $h=1,\dots, H-1$, where ${\bf{z}}_{i,-h}$ denotes the vector of binary indicators ${\bf z}_i$ without considering the $h$th one, whereas $c_{\boldsymbol{\beta}_h}$, $c_{{\sigma}^2_h}$, $c_{\boldsymbol{\alpha}_h}$, $c_{z_{ih}}$ and $c_{\omega_{ih}}$, are additive constants with respect to the argument in the corresponding variational distribution. Each expectation in the above equations is evaluated with respect to the variational distribution of the other parameters and therefore we need to rely on some iterative method to find the optimal solution. We consider the coordinate ascent variational inference (\textsc{cavi}) iterative procedure---described  in Algorithm \ref{algo3}---which maximizes the variational distribution of each parameter based on the current estimate for the remaining ones \citep[e.g.][Ch. 10]{BI06}. This procedure generates a monotonic sequence of the $\textsc{elbo}\{q_{ {\bf{x}}}(\boldsymbol{\alpha},\boldsymbol{\beta}, \boldsymbol{\sigma}^2, {\bf{z}}, \boldsymbol{\omega}) \}$, which ensures convergence to a local joint maximum. As shown in Algorithm \ref{algo3}, the normalizing constants in the above equations have not to be computed numerically, since kernels of well known distributions can be recognized.

Finally, we shall emphasize that the novel \textsc{vb} proposed in Algorithm \ref{algo3} is different from the one in \cite{RDC11}. Indeed---due to the apparent absence of conjugacy for the $\boldsymbol{\alpha}$ parameters---\cite{RDC11} rely on a connection with Bayesian hierarchical mixtures of experts \citep{BS03} and consider the lower bound of \citet{JJ00} to allow simple computation, thus providing a local \textsc{vb} routine \citep[e.g.][Ch. 10.5]{BI06}. Leveraging the conjugacy induced by the   P\'olya-gamma data augmentation, we obtain instead a global mean field \textsc{vb} routine within an exponential family representation. This guarantees that the Kullback-Leibler divergence between the variational distribution and the true posterior is minimized, and allows recent theoretical properties for this class of computational methods \citep{Blei2017} to be valid also for our variational algorithm.



\section{Epidemiology application}
\label{Section4}
We compare the performance of the three computational methods developed in Section \ref{Section3}, in a toxicology study. Consistent with recent interests in Bayesian density regression \citep[e.g.][]{DP08,HWA2014,CA17}, we focus on a dataset aimed at studying the relationship between the \texttt{DDE} concentration in maternal serum, and the gestational days at delivery \citep{LO01}. 

The \texttt{DDE} is a persistent metabolite of \textsc{ddt}, which is still used against malaria-transmitting mosquitoes in certain developing countries---according to the Malaria Report 2015 from the World Health Organization---thus raising concerns about its adverse effects on premature delivery. Popular studies in reproductive epidemiology address this goal by dichotomizing the gestational age at delivery (\texttt{GAD}) with a clinical threshold, so that births occurred before the $37$-th week are considered preterm. Although this approach allows for a simpler modeling strategy, it leads to a clear loss of information. In particular, a greater risk of mortality and morbidity is associated with preterm birth, which increases rapidly as the \texttt{GAD} decreases. This has motivated an increasing interest in modeling how the entire distribution of \texttt{GAD} changes with \texttt{DDE} exposure within a density regression framework \citep[e.g.][]{DP08,HWA2014,CA17}.

\begin{figure}[t!]
\centering
\includegraphics[width=0.47\textwidth]{plot1}
\caption{Scatterplot of the $\texttt{DDE}$ concentration against the gestational age at delivery, expressed in days, for the $2{,}312$ women in the \citet{LO01} study. The solid line is a loess estimate for the conditional mean, while the shaded area indicates $95\%$ pointwise confidence intervals. \label{Fig::dde}}
\end{figure}

\begin{figure*}[t!]
\centering
\includegraphics[width=1\textwidth]{plot2}
\caption{For selected quantiles of $\texttt{DDE} \in (12.57,28.44,53.72,105.47)$, graphical representation of the posterior mean of the conditional density for $\texttt{GAD}$ given $\texttt{DDE}$, obtained from the Gibbs sampler and the \textsc{vb}, together with $0.95$ pointwise credibility intervals (shaded area). Since the \textsc{em} provides only a mode for the conditional density, we consider a graphical representation of the plug-in estimate for  $f(y \mid x)$. The histograms represent the observations of \texttt{GAD}, having $\texttt{DDE}$ in the intervals $(-\infty,20.505)$, $[20.505,41.08)$, $[41.08,79.6)$,  $[79.6,+\infty)$, respectively.}
 \label{Fig::conditional}
\end{figure*}

As shown in Figure \ref{Fig::dde}, data are composed by $n=2{,}312$ measurements $(x_i, y_i)$, for women $i=1,\dots,n$, where $x_i$ represents the \texttt{DDE} concentration, and $y_i$ is the gestational age at delivery for woman $i$. Our goal is to reproduce the analyses in \citet{DP08} on this dataset, and compare the inference and computational performance of the \textsc{mcmc} via Gibbs sampling, the \textsc{em} algorithm, and the  \textsc{vb} routine proposed in Section \ref{Section3}. Note that, consistent with the main novelty of this contribution, we do not attempt to improve the flexibility and the efficiency of the available statistical models for Bayesian density regression---such as the kernel stick-breaking \citep{DP08}, and the \textsc{psbp} \citep{RD11}. Indeed, as discussed in Sections \ref{sec1} and \ref{Section2}, these representations are expected to provide a comparable performance to our \textsc{lsbp} in terms of inference. However, differently from current models for Bayesian density regression, inference under the  \textsc{lsbp} is available under a broader variety of computational methods, thus facilitating implementation of the same model in a wider range of applications---including large $P$ and $n$ settings. Due to this, the main focus is on providing an empirical comparison of the three algorithms proposed in Section  \ref{Section3}, while using results in \citet{DP08} as a benchmark to provide reassurance that inference under the \textsc{lsbp} is comparable to alternative representations widely considered by the practitioners.



\begin{figure*}[t!]
\centering
\includegraphics[width=0.96\textwidth]{plot3}
\caption{For the Gibbs sampler and the \textsc{vb}, posterior means of four different conditional probabilities $\mbox{pr}(y < y^{*} \mid x)$---based on thresholds $y^{*} \in (7\cdot33,7\cdot35,7\cdot37,7\cdot40)$---along with $0.95$ pointwise credibility intervals (shaded area). These quantities are not available from the \textsc{em} algorithm, for which a plug-in estimate of $\mbox{pr}(y < y^{*} \mid x)$ is displayed.}
\label{Fig::posterior}
\vspace{-5pt}
\end{figure*}
We apply the predictor--dependent mixture of Gaussians \eqref{mix_dep_gaus} with \textsc{lsbp} \eqref{logit_stick}--\eqref{pred}, to a normalized version of the \texttt{DDE} and  \texttt{GAD} $(\bar{x}_i,\bar{y}_i)$, $i=1, \ldots, n$, and then show results for $f_x(y)$ on the original scale of the data. Consistent with previous works \citep{DP08,CA17}, we let $P=2$, with $\lambda_1(\bar{x}_i)=1$ and $\lambda_2(\bar{x}_i) = \bar{x}_i$, for every $i=1, \ldots,n$, and rely instead on a flexible representation for $\eta_{h}(\bar{x}_i)$ to characterize changes in the stick-breaking weights with \texttt{DDE}. In particular, each $\eta_{h}(\bar{x}_i)$ is defined via a natural cubic spline basis $\boldsymbol{\psi}(\bar{x}_i) = \{1, \psi_1(\bar{x}_i),\dots, \psi_{5}(\bar{x}_i)\}^{\intercal}$, for every $h=1, \ldots, H-1$.  Bayesian posterior inference---under the three computational methods developed in Section~\ref{Section3}---is instead performed with default hyperparameters $\boldsymbol{\mu}_{\beta}=(0,0)^{\intercal}$, $\boldsymbol{\Sigma}_{\beta}=\mbox{I}_{2\times2}$, $\boldsymbol{\mu}_{\alpha}=(0,\dots,0)^{\intercal}$, $\boldsymbol{\Sigma}_{\alpha}=\mbox{I}_{6\times 6}$ and $a_{\sigma}=b_{\sigma}=0.1$. For the total number of mixture components we consider $H=5$, and allow the shrinkage induced by the stick-breaking prior to adaptively delete redundant components not required to characterize the data. As shown in Figure \ref{Fig::conditional}, these choices allows accurate inference on $f_x(y)$.


In providing posterior inference under the Gibbs sampling algorithm described in Section \ref{gibbs}, we rely on $30{,}000$ iterations, after discarding the first $5{,}000$ as a burn-in. Analysis of the traceplots for the quantities discussed in Figures \ref{Fig::conditional} and \ref{Fig::posterior} showed that this choice is sufficient for good convergence. The \textsc{em} algorithm and the \textsc{vb} procedures discussed in Sections \ref{em} and \ref{vb}, respectively, are instead run until convergence to a modal solution. Since such modes could be only local, we run both the algorithms for different initial values, and consider the solutions having the highest log-posterior and the lowest bound of the marginal density, respectively. We also controlled the monotonicity of the sequences for these quantities, in order to further validate the correctness of our derivations. In this study, the \textsc{em}  and the \textsc{vb} reach convergence in $2.1$ and $6.6$ seconds, respectively, whereas the Gibbs sampler requires $5$ minutes, using a MacBook Air (OS X Sierra) with a Intel Core i5.




Similarly to Figure 3 in \citet{DP08}, Figure~\ref{Fig::conditional} provides posterior inference for the conditional density $f_x(y)$ evaluated at the $0.1$, $0.6$, $0.9$, $0.99$ quantiles of \texttt{DDE}, for the three algorithms. Histograms for the \texttt{GAD}, are instead obtained by grouping the response data according to a binning of the  \texttt{DDE} with cut-offs at the central values of subsequent quantiles, so that the conditional density can be plotted alongside the corresponding histogram. Results in Figure~\ref{Fig::conditional} confirm accurate fit to the data and suggest that the left tail of the \texttt{GAD} distribution---associated with preterm deliveries---increasingly inflates as \texttt{DDE} grows. Moreover, as seen in Figure~\ref{Fig::conditional}, the three algorithms have similar results, thus providing empirical reassurance for the goodness of the proposed routines.  As expected, the point estimate from the \textsc{em} matches the posterior mean of the Gibbs sampler, whereas the \textsc{vb} tends to over--smooth some modes of the conditional distribution.  This is likely due to the fact that the \textsc{vb} outputs a mean field approximation of the posterior distribution, instead of the exact one. However, differently from the \textsc{em}, this routine allows uncertainty quantification, and provides a much scalable methodology compared to the Gibbs sampler, thus representing a valid candidate in high--dimensional inference when the focus is on specific functionals of $f_x(y)$. Indeed, as shown in Figure \ref{Fig::posterior}, when the aim is to exploit $f_x(y)$ to infer conditional preterm probabilities $$\mbox{pr}(y < y^{*} \mid x)=\int_{-\infty}^{y^{*}}f_x(y) \mbox{d}y$$
with $y^{*} \in (7{\cdot}33,7{\cdot}35,7{\cdot}37,7{\cdot}40)$ a clinical threshold, the  \textsc{vb}  provides very similar results compared to the other methods.

Prior to conclude our analysis, note that the results  in Figures \ref{Fig::conditional} and \ref{Fig::posterior} are similar to those obtained under the kernel stick-breaking prior in  \citet{DP08}. This provides empirical guarantee that the flexibility and efficiency properties characterizing popular Bayesian nonparametric models for density regression are maintained also under our \textsc{lsbp}, which has the additional relevant benefit of facilitating computational implementation of these methodologies. Minor differences are found at extreme \texttt{DDE} exposures, but this is mainly due to the sparsity of the data in this subset of the predictor space, as shown in Figure \ref{Fig::dde}.




\section{Discussion}
\label{Section5}
The focus of this paper has been on providing novel methodologies to facilitate computational implementation of Bayesian nonparametric models for density regression in a broad range of applications. To address this goal, we have proposed an alternative reparameterization of the predictor--dependent stick-breaking weights, which relies on a set of sequential logistic regressions. This constructive representation has relevant connections with continuation--ratio logistic regressions, and P\'olya-Gamma data augmentation, thus allowing simple derivation of several algorithms of routine use in Bayesian inference. The proposed computational methods are empirically evaluated in a toxicology study, obtaining good results and reassurance that the \textsc{lsbp} maintains the same  flexibility and efficiency properties characterizing popular Bayesian nonparametric models for density regression. 

Although our dependent mixture of Gaussians provides a flexible representation, it is worth considering extensions to other kernels. For example, all our algorithms can be easily adapted to predictor--independent kernels coming from an exponential family, when  conjugate priors for their parameters are used. Similar derivations are also possible for predictor--dependent kernels within a generalized linear model representation, provided that conjugate priors for the coefficients can be found \citep[e.g.][]{CHE03}. Theory and computational steps associated with the logit stick-breaking prior for the mixing probabilities are instead general and valid regardless the kernel choice.



\bibliographystyle{chicago}
\bibliography{bibliography}


\end{document}
% end of file template.tex


\begin{thebibliography}{}

\bibitem[\protect\citeauthoryear{Aitchison and Shen}{Aitchison and
  Shen}{1980}]{AS80}
Aitchison, J. and S.~M. Shen (1980).
\newblock {Logistic-normal distributions: some properties and uses}.
\newblock {\em Biometrika\/}~{\em 67\/}(2), 262--272.

\bibitem[\protect\citeauthoryear{Amemiya}{Amemiya}{1981}]{AM81}
Amemiya, T. (1981).
\newblock {Qualitative response models: a survey}.
\newblock {\em Journal of Economic Literature\/}~{\em 19\/}(4), 1483--1536.

\bibitem[\protect\citeauthoryear{Antoniano-Villalobos, Wade, and
  Walker}{Antoniano-Villalobos et~al.}{2014}]{AWW14}
Antoniano-Villalobos, I., S.~Wade, and S.~Walker (2014).
\newblock {A Bayesian nonparametric regression model with normalized weights: a
  study of hippocampal atrophy in Alzheimer's disease}.
\newblock {\em Journal of the American Statistical Association\/}~{\em
  109\/}(506), 477--490.

\bibitem[\protect\citeauthoryear{Barrientos, Jara, Quintana, et~al.}{Barrientos
  et~al.}{2012}]{BAR2012}
Barrientos, A.~F., A.~Jara, F.~A. Quintana, et~al. (2012).
\newblock On the support of maceachern’s dependent dirichlet processes and
  extensions.
\newblock {\em Bayesian Analysis\/}~{\em 7\/}(2), 277--310.

\bibitem[\protect\citeauthoryear{Billingsley}{Billingsley}{1995}]{BI94}
Billingsley, P. (1995).
\newblock {\em {Probability and Measure, Third Edition}}.
\newblock Wiley.

\bibitem[\protect\citeauthoryear{Bishop}{Bishop}{2006}]{BI06}
Bishop, C.~M. (2006).
\newblock {\em {Pattern Recognition and Machine Learning}}.
\newblock Springer.

\bibitem[\protect\citeauthoryear{Bishop and Svens{\'e}n}{Bishop and
  Svens{\'e}n}{2003}]{BS03}
Bishop, C.~M. and M.~Svens{\'e}n (2003).
\newblock {Bayesian hierarchical mixture of experts}.
\newblock In {\em {Proceedings of the Nineteenth Conference of Uncertainty of
  Artificial Intelligence}}.

\bibitem[\protect\citeauthoryear{Blei, Kucukelbir, and McAuliffe}{Blei
  et~al.}{2017}]{Blei2017}
Blei, D.~M., A.~Kucukelbir, and J.~D. McAuliffe (2017).
\newblock {Variational Inference: A Review for Statisticians}.
\newblock {\em Journal of the American Statistical Association\/}~{\em
  112\/}(518), 859--877.

\bibitem[\protect\citeauthoryear{Canale, Durante, and Dunson}{Canale
  et~al.}{2017}]{CA17}
Canale, A., D.~Durante, and D.~B. Dunson (2017).
\newblock Convex mixture regression for quantitative risk assessment.
\newblock {\em eprint arXiv:1701.02950\/}.

\bibitem[\protect\citeauthoryear{Caron, Davy, Doucet, Duflos, and
  Vanheeghe}{Caron et~al.}{2006}]{CA06}
Caron, F., M.~Davy, A.~Doucet, E.~Duflos, and P.~Vanheeghe (2006).
\newblock {Bayesian inference for dynamic models with Dirichlet process
  mixtures}.
\newblock In {\em Information Fusion, 2006 9th International Conference on},
  pp.\  1--8. IEEE.

\bibitem[\protect\citeauthoryear{Chen and Ibrahim}{Chen and
  Ibrahim}{2003}]{CHE03}
Chen, M.-H. and J.~G. Ibrahim (2003).
\newblock Conjugate priors for generalized linear models.
\newblock {\em Statistica Sinica\/}~{\em 13}, 461--476.

\bibitem[\protect\citeauthoryear{Choi and Hobert}{Choi and Hobert}{2013}]{CH13}
Choi, H.~M. and J.~P. Hobert (2013).
\newblock {The Polya-Gamma Gibbs sampler for Bayesian logistic regression is
  uniformly ergodic}.
\newblock {\em Electronic Journal of Statistics\/}~{\em 7}, 2054--2064.

\bibitem[\protect\citeauthoryear{De~Iorio, M{\"u}ller, Rosner, and
  MacEachern}{De~Iorio et~al.}{2004}]{DEI04}
De~Iorio, M., P.~M{\"u}ller, G.~L. Rosner, and S.~N. MacEachern (2004).
\newblock {An ANOVA model for dependent random measures}.
\newblock {\em Journal of the American Statistical Association\/}~{\em
  99\/}(465), 205--215.

\bibitem[\protect\citeauthoryear{{De la Cruz-Mes{\'\i}a}, Quintana, and
  M{\"u}ller}{{De la Cruz-Mes{\'\i}a} et~al.}{2007}]{CRU07}
{De la Cruz-Mes{\'\i}a}, R., F.~A. Quintana, and P.~M{\"u}ller (2007).
\newblock Semiparametric {Bayesian} classification with longitudinal markers.
\newblock {\em Journal of the Royal Statistical Society: Series C (Applied
  Statistics)\/}~{\em 56\/}(2), 119--137.

\bibitem[\protect\citeauthoryear{Dempster, Laird, and Rubin}{Dempster
  et~al.}{1977}]{DL77}
Dempster, A.~P., N.~M. Laird, and D.~B. Rubin (1977).
\newblock {Maximum likelihood from incomplete data via the {EM} algorithm}.
\newblock {\em Journal of the Royal Statistical Society, Series B\/}~{\em
  39\/}(1), 1--38.

\bibitem[\protect\citeauthoryear{Dunson and Park}{Dunson and Park}{2008}]{DP08}
Dunson, D.~B. and J.~H. Park (2008).
\newblock {Kernel stick-breaking processes}.
\newblock {\em Biometrika\/}~{\em 95\/}(2), 307--323.

\bibitem[\protect\citeauthoryear{Escobar and West}{Escobar and
  West}{1995}]{EW95}
Escobar, M.~D. and M.~West (1995).
\newblock {Bayesian} density estimation and inference using mixtures.
\newblock {\em Journal of the American Statistical Association\/}~{\em
  90\/}(430), 577--588.

\bibitem[\protect\citeauthoryear{Ferguson}{Ferguson}{1973}]{FE73}
Ferguson, T.~S. (1973).
\newblock {A Bayesian analysis of some nonparametric problems}.
\newblock {\em The Annals of Statistics\/}~{\em 1}, 209--230.

\bibitem[\protect\citeauthoryear{Gelfand, Kottas, and MacEachern}{Gelfand
  et~al.}{2005}]{GEL05}
Gelfand, A.~E., A.~Kottas, and S.~N. MacEachern (2005).
\newblock {Bayesian} nonparametric spatial modeling with {Dirichlet} process
  mixing.
\newblock {\em Journal of the American Statistical Association\/}~{\em
  100\/}(471), 1021--1035.

\bibitem[\protect\citeauthoryear{Ghosal, Ghosh, and Ramamoorthi}{Ghosal
  et~al.}{1999}]{GHO99}
Ghosal, S., J.~K. Ghosh, and R.~Ramamoorthi (1999).
\newblock Posterior consistency of {Dirichlet} mixtures in density estimation.
\newblock {\em The Annals of Statistics\/}~{\em 27\/}(1), 143--158.

\bibitem[\protect\citeauthoryear{Ghosal and Van Der~Vaart}{Ghosal and Van
  Der~Vaart}{2007}]{GHO07}
Ghosal, S. and A.~Van Der~Vaart (2007).
\newblock Posterior convergence rates of {Dirichlet} mixtures at smooth
  densities.
\newblock {\em The Annals of Statistics\/}~{\em 35\/}(2), 697--723.

\bibitem[\protect\citeauthoryear{Griffin and Steel}{Griffin and
  Steel}{2006}]{GS06}
Griffin, J.~E. and M.~Steel (2006).
\newblock {Order-based dependent Dirichlet processes}.
\newblock {\em Journal of the American Statistical Association\/}~{\em 10},
  179--194.

\bibitem[\protect\citeauthoryear{Griffin and Steel}{Griffin and
  Steel}{2011}]{GRI11}
Griffin, J.~E. and M.~F. Steel (2011).
\newblock Stick-breaking autoregressive processes.
\newblock {\em Journal of Econometrics\/}~{\em 162\/}(2), 383--396.

\bibitem[\protect\citeauthoryear{Guti{\'e}rrez, Mena, and
  Ruggiero}{Guti{\'e}rrez et~al.}{2016}]{GUT16}
Guti{\'e}rrez, L., R.~H. Mena, and M.~Ruggiero (2016).
\newblock A time dependent {Bayesian} nonparametric model for air quality
  analysis.
\newblock {\em Computational Statistics \& Data Analysis\/}~{\em 95}, 161--175.

\bibitem[\protect\citeauthoryear{Hwang and Pennell}{Hwang and
  Pennell}{2014}]{HWA2014}
Hwang, B.~S. and M.~L. Pennell (2014).
\newblock Semiparametric bayesian joint modeling of a binary and continuous
  outcome with applications in toxicological risk assessment.
\newblock {\em Statistics in Medicine\/}~{\em 33\/}(7), 1162--1175.

\bibitem[\protect\citeauthoryear{Ishwaran and James}{Ishwaran and
  James}{2001}]{IJ01}
Ishwaran, H. and L.~F. James (2001).
\newblock {Gibbs sampling methods for stick-breaking priors}.
\newblock {\em Journal of the American Statistical Association\/}~{\em 96},
  161--173.

\bibitem[\protect\citeauthoryear{Ishwaran and James}{Ishwaran and
  James}{2002}]{IJ02}
Ishwaran, H. and L.~F. James (2002).
\newblock {Approximate Dirichlet process computing finite normal mixtures:
  smoothing and prior information}.
\newblock {\em Journal of Computational and Graphical Statistics\/}~{\em
  11\/}(3), 508--532.

\bibitem[\protect\citeauthoryear{Jaakkola and Jordan}{Jaakkola and
  Jordan}{2000}]{JJ00}
Jaakkola, T.~S. and M.~I. Jordan (2000).
\newblock {Bayesian parameter estimation via variational methods}.
\newblock {\em Statistics and Computing\/}~{\em 10}, 25--37.

\bibitem[\protect\citeauthoryear{Jordan and Jacobs}{Jordan and
  Jacobs}{1994}]{JJ94}
Jordan, M.~I. and R.~A. Jacobs (1994).
\newblock {Hierarchical mixture of experts and the EM algorithm}.
\newblock {\em Neural Computation\/}~{\em 6}, 181 -- 214.

\bibitem[\protect\citeauthoryear{Kalli, Griffin, and Walker}{Kalli
  et~al.}{2011}]{KGW11}
Kalli, M., J.~E. Griffin, and S.~G. Walker (2011, January).
\newblock {Slice sampling mixture models}.
\newblock {\em Statistics and Computing\/}~{\em 21\/}(1), 93--105.

\bibitem[\protect\citeauthoryear{Linderman, Johnson, and Adams}{Linderman
  et~al.}{2015}]{LIND2015}
Linderman, S., M.~Johnson, and R.~P. Adams (2015).
\newblock Dependent multinomial models made easy: Stick-breaking with the
  p{\'o}lya-gamma augmentation.
\newblock In {\em Advances in Neural Information Processing Systems},
  Volume~28, pp.\  3456--3464.

\bibitem[\protect\citeauthoryear{Longnecker, Klebanoff, Zhou, and
  Brock}{Longnecker et~al.}{2001}]{LO01}
Longnecker, M.~P., M.~A. Klebanoff, H.~Zhou, and J.~W. Brock (2001).
\newblock Association between maternal serum concentration of the ddt
  metabolite dde and preterm and small-for-gestational-age babies at birth.
\newblock {\em Lancet\/}~{\em 358}, 110--114.

\bibitem[\protect\citeauthoryear{MacEachern}{MacEachern}{1999}]{MC99}
MacEachern, S.~N. (1999).
\newblock {Dependent nonparametric processes}.
\newblock In {\em {Proceedings of the Bayesian Section}}, {Alexandria, VA:
  American Statistical Association}, pp.\  50--55.

\bibitem[\protect\citeauthoryear{MacEachern}{MacEachern}{2000}]{MC00}
MacEachern, S.~N. (2000).
\newblock {Dependent Dirichlet processes}.
\newblock Technical report, Department of Statistics, Ohio State University.

\bibitem[\protect\citeauthoryear{Meng and Rubin}{Meng and Rubin}{1993}]{MR93}
Meng, X.-L. and D.~B. Rubin (1993).
\newblock {Maximum likelihood estimation via the ECM algorithm: a general
  framework}.
\newblock {\em Biometrika\/}~{\em 80\/}(2), 267--278.

\bibitem[\protect\citeauthoryear{Neal}{Neal}{2000}]{NEA00}
Neal, R.~M. (2000).
\newblock Markov chain sampling methods for {Dirichlet} process mixture models.
\newblock {\em Journal of Computational and Graphical Statistics\/}~{\em
  9\/}(2), 249--265.

\bibitem[\protect\citeauthoryear{Nelder and Wedderburn}{Nelder and
  Wedderburn}{1972}]{NW72}
Nelder, J.~A. and R.~W.~M. Wedderburn (1972).
\newblock {Generalized linear models}.
\newblock {\em Journal of the Royal Statistical Society, Series A\/}~{\em
  135\/}(3), 370 -- 384.

\bibitem[\protect\citeauthoryear{Pati, Dunson, and Tokdar}{Pati
  et~al.}{2013}]{PA013}
Pati, D., D.~B. Dunson, and S.~T. Tokdar (2013).
\newblock Posterior consistency in conditional distribution estimation.
\newblock {\em Journal of Multivariate Analysis\/}~{\em 116}, 456--472.

\bibitem[\protect\citeauthoryear{Pitman and Yor}{Pitman and Yor}{1997}]{PY97}
Pitman, J. and M.~Yor (1997).
\newblock {The two-parameter Poisson-Dirichlet distribution derived from a
  stable subordinator}.
\newblock {\em The Annals of Probability\/}~{\em 25\/}(2), 855--900.

\bibitem[\protect\citeauthoryear{Polson, Scott, and Windle}{Polson
  et~al.}{2013}]{PS13}
Polson, N.~G., J.~G. Scott, and J.~Windle (2013).
\newblock {Bayesian inference for logistic models using P{\'o}lya--Gamma latent
  variables}.
\newblock {\em Journal of the American Statistical Association\/}~{\em
  108\/}(504), 1339--1349.

\bibitem[\protect\citeauthoryear{Rasmussen and Williams}{Rasmussen and
  Williams}{2006}]{RW06}
Rasmussen, C. and C.~Williams (2006).
\newblock {\em {Gaussian Processes for Machine Learning}}.
\newblock The MIT Press.

\bibitem[\protect\citeauthoryear{Ren, Du, Carin, and Dunson}{Ren
  et~al.}{2011}]{RDC11}
Ren, L., L.~Du, L.~Carin, and D.~B. Dunson (2011).
\newblock {Logistic stick-breaking process}.
\newblock {\em Journal of Machine Learning Research\/}~{\em 12}, 203--239.

\bibitem[\protect\citeauthoryear{Rodriguez and Dunson}{Rodriguez and
  Dunson}{2011}]{RD11}
Rodriguez, A. and D.~B. Dunson (2011).
\newblock {Nonparametric Bayesian models through probit stick-breaking
  processes}.
\newblock {\em Bayesian Analysis\/}~{\em 6\/}(1), 145--178.

\bibitem[\protect\citeauthoryear{Sethuraman}{Sethuraman}{1994}]{SE94}
Sethuraman, J. (1994).
\newblock {A constructive definition of Dirichlet priors}.
\newblock {\em Statistica Sinica\/}~{\em 4}, 639--650.

\bibitem[\protect\citeauthoryear{Tokdar}{Tokdar}{2006}]{TOK06}
Tokdar, S.~T. (2006).
\newblock Posterior consistency of dirichlet location-scale mixture of normals
  in density estimation and regression.
\newblock {\em Sankhy{\=a}: The Indian Journal of Statistics\/}, 90--110.

\bibitem[\protect\citeauthoryear{Tutz}{Tutz}{1991}]{TU91}
Tutz, G. (1991).
\newblock {Sequential models in categorical regression}.
\newblock {\em Computational Statistics \& Data Analysis\/}~{\em 11}, 275--295.

\bibitem[\protect\citeauthoryear{Wade, Dunson, Petrone, and Trippa}{Wade
  et~al.}{2014}]{WDPT14}
Wade, S., D.~B. Dunson, S.~Petrone, and L.~Trippa (2014).
\newblock {Improving Prediction from Dirichlet Process Mixtures via
  Enrichment}.
\newblock {\em Journal of Machine Learning Research\/}~{\em 15}, 1041--1071.

\bibitem[\protect\citeauthoryear{Walker}{Walker}{2007}]{WA07}
Walker, S.~G. (2007).
\newblock {Sampling the Dirichlet mixture model with slices}.
\newblock {\em Communications in Statistics -- Simulation and
  Computation\/}~{\em 36}, 45--54.

\end{thebibliography}