Nonparametric Bayesian Inference for Partially Identified Discrete Response ModelsThanks: August 26, 2026. Tamer: Harvard University, Department of Economics, elietamer@fas.harvard.edu; Walker: Duke University, Department of Economics, christopher.walker@duke.edu. This work is preliminary and any comments are welcome. We are grateful to Aslihan Asil, Allan Collard-Wexler, and Adam Rosen for helpful discussions. We thank Mengyuan Jiang for research assistance. All errors are our own.

Many economic models with discrete responses lead to set-identified structural parameters. These include games with multiple equilibria (Tamer, 2003; Ciliberto and Tamer, 2009), semiparametric static binary/multinomial choice models with fixed effects (Manski, 1987; Shi et al., 2018; Gao and Li, 2024; Pakes and Porter, 2024), dynamic models with state dependence (Heckman, 1981; Honoré and Tamer, 2006; Torgovitsky, 2019; Khan et al., 2021; Pakes et al., 2021; Khan et al., 2023), limited consideration discrete choice models (Barseghyan et al., 2021; Lu, 2022), ordered choice models (Chesher and Smolinski, 2012; Pakes et al., 2015), network formation models (Sheng, 2020), and binary regression models with interval-censored covariates (Manski and Tamer, 2002), to name a few. A common statistical theme across these examples is that the identified set is known up to an unrestricted reduced-form conditional probability mass function (PMF). Consequently, uncertainty about the identified set entirely reflects that of the PMF, and, as a result, statistical inference for the PMF should translate to the identified set.

Using this observation, our paper proposes a nonparametric Bayesian inference framework for identified sets defined by conditional PMFs. Concretely, let the observed data be $(Y,X^{\prime})^{\prime}$, where $Y$ is a discrete random variable and $X$ is a (possibly) continuous random vector, and let the target parameter be an identified set $\Gamma_{I}=\{\gamma:f(X,p(X),\gamma)\geq 0\ a.s.\}$, where $\gamma$ is a finite-dimensional economic parameter (e.g., payoff parameters), $p(X)$ is a reduced-form conditional PMF for $Y$ given $X$, and $f(X,p(X),\gamma)$ is a known vector of possibly nonseparable functions (with inequalities evaluated elementwise). The central idea of our proposal is that $\Gamma_{I}$ can be viewed as the output of a correspondence that takes $p$ as an input. Consequently, a Bayesian inference framework for $\Gamma_{I}$ arises in which a marginal posterior $\Gamma_{I}|Y,X$ is obtained from a posterior $p|Y,X$ for $p$. Beyond this, we provide implementation details for Gaussian process (GP) priors, prove a general posterior consistency theorem for $\Gamma_{I}|Y,X$ (and for a pseudo-identified set that accommodates misspecification), and verify the conditions of the theorem for GPs. The examples below highlight the breadth of our framework; Section 5 also demonstrates how many of our results extend to continuous $Y$ and aggregated discrete choice (e.g., $Y$ is a market share).

Our proposal has several appealing features. First, our approach accommodates continuous $X$ without requiring the conversion of conditional moments into unconditional moments or explicitly discretizing the support of $X$ into bins. In CMI models, the one-sided nature of inequality restrictions means that making these choices in a way that preserves identifying information is challenging (Khan and Tamer, 2009; Andrews and Shi, 2013). This has resulted in a widespread empirical practice of basing inference on ad hoc choices. For instance, in many empirical implementations of CMIs, the conditioning variables are discretized into finite cells or groups. The resulting cell-level inequalities can then be treated as a finite set of unconditional moment inequalities, typically by multiplying the conditional moment by cell indicators or other nonnegative ‘instruments’ and averaging (see, for example, Kline and Tamer (2016); Chen et al. (2018); Ciliberto et al. (2021); Pakes et al. (2021), among many others). Ad hoc selections of unconditional moments may result in substantial loss of sharpness under correct specification, while conclusions drawn from outer sets based on unconditional moments may be sensitive to instrument choice when the underyling CMIs are misspecified (Li et al., 2024). Similarly, empirical applications of linear systems typically involve coarsened covariates. For instance, the linear programming formulation Honoré and Tamer (2006) presumes that the covariates have finite support (so as to obtain a finite set of inequalities), while the empirical implementation of the identified set in Khan et al. (2023) binarizes the covariates, despite the finding that rich covariate support can lead to much sharper identified sets (and even point identification in some cases). Similar discretizing steps were taken in Ciliberto and Tamer (2009) and Kline and Tamer (2016). In both CMIs and linear systems, our procedure obviates the need for such discretization and/or selecting unconditional moments by targeting the conditional PMF directly.

Second, our approach is flexible and amenable to computation. We provide implementation details for a class of priors that combine stick-breaking and Gaussian process (GP) methods.²² 2 Stick-breaking is a technique that dates back to Halmos (1944) and can be thought of as representing a discrete distribution in terms of its hazard functions. It was famously employed by Sethuraman (1994) as a computational device for the Ferguson (1973) Dirichlet process, however these techniques are increasingly used for Bayesian conditional distribution estimation (Dunson and Park, 2008; Chung and Dunson, 2009; Rodrıguez and Dunson, 2011; Ren et al., 2011). GP priors allow a researcher to flexibly incorporate nonparametric properties, such as smoothness, and produce nonparametric Bayesian procedures with similar flexibility to frequentist nonparametric estimators (van der Vaart and van Zanten, 2008a). Consequently, our implementation shares the advantageous flexibility of frequentist inference for identified sets, while enjoying all of the niceties of Bayesian inference. These priors are also computationally attractive because the stick-breaking characterization enables the use of the Pólya-Gamma data augmentation technique of Polson et al. (2013) to derive Gibbs sampling algorithms in which GPs form conditionally conjugate priors. As a result, sampling from the posterior for the reduced-form PMFs amounts to generating draws from standard parametric distributions. Given draws from $p|Y,X$, the researcher can use any appropriate method for computing $\Gamma_{I}$ to obtain the corresponding draws from $\Gamma_{I}|Y,X$, while the two-step nature of our proposal means that computing the $\Gamma_{I}$ draws is parallelizable.

The main theoretical results in the paper are posterior consistency guarantees for the identified set. A possible concern about Bayesian inference for partially identified models is that, even with infinite data, the prior does not fully revise (Poirier, 1998; Moon and Schorfheide, 2012; Giacomini and Kitagawa, 2021). In our setting, the prior is placed directly on an identifiable reduced-form parameter $p$ and the identified set $\Gamma_{I}$ is viewed as a point identified, set-valued parameter. Consequently, under suitable continuity restrictions on the mapping from reduced form PMFs to identified sets, fully revised beliefs for the PMF should translate to fully revised beliefs about the identified set. Our core posterior consistency theorem confirms this intuition, providing high-level conditions under which posterior consistency for the conditional choice probabilities leads to the posterior for the identified set to concentrate around the true identified set. In this sense, the prior for $p$ has an asymptotically negligible effect on inferences about $\Gamma_{I}$. We also prove posterior consistency for a pseudo-identified set $\tilde{\Gamma}_{I}$ that is always nonempty (and coincides with $\Gamma_{I}$ when $\Gamma_{I}$ is not empty), and show that the posterior probability that the identified set is empty is a consistent diagnostic for model misspecification. This is important because several of the examples that fit within our framework are prone to model misspecification, for instance, due to parametric restrictions on unobserved heterogeneity (Ciliberto and Tamer, 2009; Barseghyan et al., 2021; Ciliberto et al., 2021).

Our posterior consistency theorems are based on assumptions that the posterior concentrates on sets of $p$ for which certain uniform convergence and lower hemicontinuity conditions are satisfied. While these conditions are straightforward to verify in finite-dimensional models, posterior consistency in infinite-dimensional models can be a delicate issue, evidenced by canonical inconsistency results, such as those in Freedman (1963); Freedman (1965) and Diaconis and Freedman (1986a); Diaconis and Freedman (1986b); Diaconis and Freedman (1993), and more recent work highlighting the intricacies of posterior consistency in norms relevant to statistical functionals (Giné and Nickl, 2011; Castillo, 2014; Ho and Walker, 2024). Since we cast the identified set as a statistical functional, this second set of concerns is relevant for our framework. Conscious of this, we enhance the credibility of our general theory with a set low-level sufficient conditions for an important class of identified sets (i.e., they apply to linear systems and some conditional moment inequality models) under both correct specification (i.e., $\Gamma_{I}$) and misspecification (i.e., $\tilde{\Gamma}_{I}$). Then, noticing a common underlying supremum norm posterior consistency requirement for $p$, we derive conditions under which the stick-breaking GP prior used for the implementation meets this condition, and thereby provides a complete verification of posterior consistency. As an intermediate technical result, we find conditions under which the stick-breaking GP posterior for $p$ converges to the truth $p_{0}$ in empirical mean-square at the minimax optimal rate, which may be of independent interest.

This paper is related to several literatures in econometrics. The first is Bayesian analysis of partially identified models. We build on Kline and Tamer (2016) by performing Bayesian inference for an identified set by evaluating it at draws from the posterior of an identifiable reduced-form parameter (i.e., the PMF). A key difference, however, is that Kline and Tamer (2016) explicitly restrict the reduced-form parameter to be finite-dimensional, thereby ruling out partially identified discrete response models with continuous covariates. Liao and Simoni (2019) and Florens and Simoni (2021) also derive reduced-form Bayesian inference frameworks for identified sets using support functions and Dirichlet processes, respectively. These papers similarly restrict attention to reduced-form parameters that are estimable at the parametric rate, precluding settings where the identified set is indexed by a reduced-form conditional PMF with continuous conditioning variables. Norets and Tang (2014) propose Bayesian inference for Rust (1987)-type structural dynamic discrete choice models by assuming a joint prior for the reduced-form conditional choice probabilities and observable state transition distributions, but restrict attention to finite observable state spaces. More recently, taking a posterior for an identified set as given, Kline and Tamer (2024) propose a Bayesian inference framework for counterfactual prediction sets, and establish continuous mapping theorems under which posterior consistency for the identified set implies predictive consistency for the counterfactuals. Since our posterior consistency results for $\Gamma_{I}$ match those required by Kline and Tamer (2024), an implication of our theory is posterior consistency for counterfactual prediction sets without requiring discretization of covariates. Other papers on Bayesian inference under partial identification include Poirier (1998), Liao and Jiang (2010), Moon and Schorfheide (2012), Giacomini and Kitagawa (2021), and Christensen et al. (2026).³³ 3 There is also a related literature in statistics on nonparametric Bayesian level set estimation. See for example Gayraud and Rousseau (2005); Gayraud and Rousseau (2007)’s work on density level sets and Li and Ghosal (2021) who focus on level sets of a general unknown function.

More broadly, our paper fits within the literature on reduced-form Bayesian inference for econometric models, which refers to situations in which a parameter of interest is viewed as a functional of an identifiable reduced-form parameter (for which a prior is assumed).⁴⁴ 4 The reduced-form Bayesian literature can be connected to a large literature in statistics on Bayesian estimation of statistical functionals. A nonexhaustive list of contributions includes Ferguson (1973), Rubin (1981), Newton and Raftery (1994), Rivoirard and Rousseau (2012), Castillo and Rousseau (2015), Lyddon et al. (2019), Nickl and Ray (2020), Monard et al. (2021), Ray and van der Vaart (2021), and Li et al. (2026). Walker (2026) considers Bayesian inference for parameters point identified by conditional moment equalities via a nonparametric prior for the conditional density of the endogenous variables given the exogenous variables. Our paper follows a similar thought experiment, except that it assumes a nonparametric prior for a conditional PMF and focuses on identified sets determined by inequality restrictions on the conditional distribution. In this sense, the conceptual relationship between our paper and Walker (2026) parallels how Kline and Tamer (2016) connects the seminal inference framework of Chamberlain and Imbens (2003) to identified sets. Norets and Pelenis (2022a); Norets and Pelenis (2022b) develop a nonparametric Bayesian frameworks for reduced-form mixed continuous-discrete distributions, with a key empirical motivation being that firm entry parameters (e.g., Pakes et al. (2007)) are functionals of these distributions. Ray and van der Vaart (2020), Breunig et al. (2025); Breunig et al. (2026), DiTraglia and Liu (2025), Yiu et al. (2025), and Ye (2026) propose reduced-form Bayesian approaches for treatment effect parameters.

Finally, our paper contributes to a large literature on inference under partial identification. Since we characterize identified sets as level sets of criterion functions, our approach offers a Bayesian analogue to frequentist inference procedures proposed in Manski and Tamer (2002), Chernozhukov et al. (2007), Bugni (2010), Romano and Shaikh (2010), Menzel (2014), Chernozhukov et al. (2015), and Chen et al. (2018). Our two leading examples, conditional moment inequalities and linear systems, connects our Bayesian framework to frequentist inference frameworks for these models, such as Khan and Tamer (2009), Andrews and Shi (2013); Andrews and Shi (2014); Andrews and Shi (2017), Chernozhukov et al. (2013), Armstrong (2014); Armstrong (2015), Armstrong and Chan (2016), Chetverikov (2018), Andrews et al. (2023), and Chernozhukov et al. (2023) for conditional moment inequalities, and Bai et al. (2022), Fang et al. (2023), Goff and Mbakop (2025), and Bai et al. (2026) for linear systems. There are many other frequentist approaches to inference under partial identification; see Canay and Shaikh (2017), Ho and Rosen (2017), Molinari (2020), Kline et al. (2021), and Canay et al. (2023) and the references therein.

The paper is organized as follows. Section 2 formalizes the Bayesian inference framework, Section 3 puts forward a flexible class of priors for $p$ and illustrates their use in an application to state dependence, Section 4 presents our posterior consistency results, Section 5 offers several extensions of the framework, and Section 6 concludes. Notation is introduced when appropriate and proofs are in the Appendix.

Let $\{(Y_{i},X_{i}^{\prime})^{\prime}\}_{i\geq 1}$, where $Y_{i}\in\mathcal{Y}$, $\mathcal{Y}=\{y_{1},...,y_{K}\}$, and $X_{i}\in\mathcal{X}\subseteq\mathbb{R}^{d_{x}}$ for each $i$, be a sequence of random vectors for which $\{(Y_{i},X_{i}^{\prime})^{\prime}\}_{i=1}^{n}$, $n\geq 1$, forms the observed data. There are no restrictions on $\mathcal{X}$ (i.e., $X_{i}$ can have continuous components). Our sampling model conditions on the realization $\{x_{i}\}_{i\geq 1}$ of $\{X_{i}\}_{i\geq 1}$ and imposes that $Y^{(n)}=(Y_{1},...,Y_{n})$ satisfies

where $Categorical(p(x))$ denotes a discrete distribution over $\{y_{1},...,y_{K}\}$ with probability mass function $p(x)=(p_{1}(x),...,p_{K}(x))$ for $x\in\mathcal{X}$. For Bayesian inference, we endow the infinite-dimensional space $\mathcal{P}$ of PMFs $p$ with a prior $\Pi$, a conditional (on $\{X_{i}\}_{i\geq 1}$) probability measure over $(\mathcal{P},\mathscr{P})$, where $\mathscr{P}$ is a $\sigma$-algebra over $\mathcal{P}$. Section 3 presents an example.

Assumptions 1 and 2 are standard regularity conditions (Ghosal and van der Vaart, 2017), and imply that the vector of in-sample PMFs $(p(x_{1}),...,p(x_{n}))$ has a well-defined posterior distribution $\Pi_{n}((p(x_{1}),...,p(x_{n}))\in\cdot|Y^{(n)})$ with density that satisfies

for each $p\in\mathcal{P}$. The posterior contains a full inference theory for the vector $(p(x_{1}),...,p(x_{n}))$ and any transformation of it. Consequently, by treating the identified set as a function of this vector, we obtain a Bayesian inference theory for the identified set.

We now show how $\pi_{n}(\cdot|Y^{(n)})$ leads to a Bayesian inference framework for an identified set. Let $\Gamma\subseteq\mathbb{R}^{d_{\gamma}}$, $d_{\gamma}<\infty$, be the structural parameter space, and, for a fixed realization $\{x_{i}\}_{i\geq 1}$ of $\{X_{i}\}_{i\geq 1}$, define the conditional identified set $\Gamma_{n,I}(p)$ at $p$ as

where $f:\mathcal{X}\times\Delta^{K-1}\times\Gamma\rightarrow\mathbb{R}^{d_{f}}$, $d_{f}<\infty$, is a known function.⁵⁵ 5 Theorem 3 resolves the apparent discrepancy between $\Gamma_{n,I}$ and $\Gamma_{I}$. The conditional identified set can be equivalently expressed as the level set $\Gamma_{n,I}(p)=\{\gamma\in\Gamma:Q_{n}(\gamma,p)=0\}$, where $Q_{n}:\Gamma\times\mathcal{P}\rightarrow\mathbb{R}_{+}$ is any criterion function that only depends on $p$ through $(p(x_{1}),...,p(x_{n}))$, and, importantly, satisfies the property⁶⁶ 6 Section 2.3 demonstrates $Q_{n}$ may depend on $\{x_{i}\}_{i=1}^{n}$, but we suppress for ease of exposition.

Crucially, since $Q_{n}$ only depends on the in-sample response probabilities $(p(x_{1}),...,p(x_{n}))$, posterior probability statements about $(p(x_{1}),...,p(x_{n}))$ translate to statements about $\Gamma_{n,I}(p)$, provided that the correspondence $\Gamma_{n,I}:\mathcal{P}\rightarrow 2^{\Gamma}$ is suitably measurable.

For this purpose, we maintain that $\Gamma_{n,I}:\mathcal{P}\rightarrow 2^{\Gamma}$ forms a random closed set (Molchanov, 2017; Molchanov and Molinari, 2018). That is, $\Gamma_{n,I}(p)$ is a closed set for each $p$ and $\Gamma_{n,I}^{-}(A):=\{p:\Gamma_{n,I}(p)\cap A\neq\emptyset\}\in\mathscr{P}$ for each compact subset $A$ of $\Gamma$. In such case, the probability law of $\Gamma_{n,I}|Y^{(n)}$ is uniquely determined by the posterior capacity functional $T_{\Gamma_{n,I}|Y^{(n)}}$, which satisfies $T_{\Gamma_{n,I}|Y^{(n)}}(A)=\Pi_{n}(\Gamma_{n,I}(p)\cap A\neq\emptyset|Y^{(n)})$ for $A\subseteq\Gamma$ compact. The posterior capacity functional formalizes $\Gamma_{I}|Y,X$. Assumptions 3 and 4, both standard, are sufficient for this characterization (with Theorem 1 serving as validation).⁷⁷ 7 Two comments. First, compactness of $\Gamma$ is not used in the proof of Theorem 1, however, for the application of random set theory, the ambient space (i.e., $\Gamma$) is typically restricted to be locally compact second countable Hausdorff (LCSCH) (see Molchanov (2017); Molchanov and Molinari (2018)). Assumption 3 is sufficient for this and will be used in the subsequent results of the paper, so we impose it outright but explicitly acknowledge that it is not necessary for Theorem 1 (i.e, we can impose that $\Gamma$ is LCSCH only). Second, if Assumption 4.1 is strengthened to continuity and Assumption 3 holds, then Assumption 4.2 holds if $Q_{n}(\gamma,\cdot):\mathcal{P}\rightarrow\mathbb{R}_{+}$ is $\mathscr{P}$-measurable for each $\gamma\in\Gamma$ (see, for example, Theorem 17.18 in Aliprantis and Border (1999)). For sufficient conditions based on lower semicontinuity, see Stinchcombe and White (1992).

There are several aspects of our framework that should be emphasized. First, the posterior for the identified set implies a marginal posterior for the identified set of functions of the structural parameter. Let $g:\mathbb{R}^{d_{\gamma}}\rightarrow\mathbb{R}^{d_{h}}$, $d_{g}<\infty$, be a known continuous transformation. The conditional identified set for $g(\gamma)$ is $G_{n,I}(p)=\{g(\gamma):\gamma\in\Gamma_{n,I}(p)\}$. Corollary 1 shows that $G_{n,I}:\mathcal{P}\rightarrow 2^{g(\Gamma)}$ forms a random closed set. Consequently, $\pi_{n}(\cdot|Y^{(n)})$ leads to a posterior for $G_{n,I}(p)$ that is similarly described its capacity functional. Defining $g$ as the coordinate projection, this nests the identified set for subvectors of $\gamma$.

Second, our framework also accommodates model misspecification. A partially identified model is misspecified at $p\in\mathcal{P}$ if $\Gamma_{n,I}(p)=\emptyset$. Setting $A=\Gamma$, $\{p:\Gamma_{n,I}(p)=\emptyset\}$ is $\mathscr{P}$-measurable and $1-T_{\Gamma_{n,I}|Y^{(n)}}(\Gamma)$ is a misspecification diagnostic because a high posterior probability of an empty identified set provides strong evidence that the identifying restrictions are not compatible with the data. Alternatively, one can report the posterior for a pseudo-identified set $\tilde{\Gamma}_{n,I}(p)=\{\gamma\in\Gamma:\tilde{Q}_{n}(\gamma,p)=0\}$, where $\tilde{Q}_{n}(\gamma,p)=Q_{n}(\gamma,p)-\inf_{\tilde{\gamma}\in\Gamma}Q_{n}(\tilde{\gamma},p)$ (i.e., $\tilde{\Gamma}_{n,I}(p)$ is the set of minimizers of $Q_{n}(\cdot,p)$). Like $\Gamma_{n,I}(p)$, $\tilde{\Gamma}_{n,I}(p)$ is a set-valued functional of $(p(x_{1}),...,p(x_{n}))$ so it has a posterior under $\pi_{n}(\cdot|Y^{(n)})$ as long as it is a random closed set.⁹⁹ 9 There are, of course, conceptual issues associated with reporting pseudo-true parameters (White, 1982; Müller, 2013; Hansen and Lee, 2021; Andrews et al., 2026). Our paper does not take a stance on whether one should report $\tilde{\Gamma}_{n,I}(p)$, but simply provides it as an option for those concerned about misspecification at $p$. Similarly, a valid inference framework for functions of $\gamma$ is obtained via the posterior for $\tilde{G}_{n,I}(p)=\{g(\gamma):\gamma\in\tilde{\Gamma}_{n,I}(p)\}$.

Finally, most literature on partial identification with covariates assumes that $\{(Y_{i},X_{i}^{\prime})^{\prime}\}_{i\geq 1}$ forms an independent and identically distributed (i.i.d) sequence, and defines the identified set at $p$ as $\Gamma_{I}(p)=\{\gamma\in\Gamma:f(x,p(x),\gamma)\geq 0\ \forall\ x\in\mathcal{S}_{X}\}$ to define the identified set at $p$, where $\mathcal{S}_{X}$ is the support of the marginal distribution $P_{0,X}$ of $X$ (see, for example, Ho and Rosen (2017), Molinari (2020), Kline et al. (2021), and Canay et al. (2023)). This creates an apparent discrepancy between our target parameter $\Gamma_{n,I}(p)$ and the focus of much of the partial identification literature (an exception being Rosen and Ura (2025), who define the identified set for the Manski (1975); Manski (1985) maximum score model conditionally). Fortunately, the next result establishes that, under a well-separatedness condition for $\Gamma_{I}(p)$ and other standard regularity conditions, there is virtually no difference between $\Gamma_{n,I}(p)$ and $\Gamma_{I}(p)$ for large $n$. For notation, $f_{j}(x,p(x),\gamma)$ is the $j$th element of $f(x,p(x),\gamma)$, $\mathscr{X}$ is the $\sigma$-algebra over $\mathcal{X}$ for which $P_{0,X}$ is defined, $\mathscr{X}^{\infty}$ is the associated product $\sigma$-algebra, and $d_{\mathcal{H}}$ is the Hausdorff distance.¹⁰¹⁰ 10 Given two sets $\Gamma_{1},\Gamma_{2}$ in $(\mathbb{R}^{d_{\gamma}},||\cdot||_{2})$, the Hausdorff distance is defined as $d_{\mathcal{H}}(\Gamma_{1},\Gamma_{2})=\max\{\sup_{\gamma\in\Gamma_{1}}\inf_{\tilde{\gamma}\in\Gamma_{2}}||\gamma-\tilde{\gamma}||_{2},\sup_{\gamma\in\Gamma_{2}}\inf_{\tilde{\gamma}\in\Gamma_{1}}||\gamma-\tilde{\gamma}||_{2}\}$.

This section revisits Examples 1 and 2. Let $||\cdot||_{n,r}$ satisfy $||h||_{n,r}=(n^{-1}\sum_{i=1}^{n}|h(x_{i})|^{r})^{1/r}$ for $r\in[1,\infty)$ and $||h||_{n,\infty}=\max_{1\leq i\leq n}|h(x_{i})|$ for real-valued $h$ (replacing $|h(x)|$ with $||h(x)||_{2}$ and $||h(x)||_{op}$ for vector valued and matrix valued $h$, respectively, where $||\cdot||_{op}$ is the operator norm). Let $\text{dist}_{W}(\cdot,A)$ satisfy $\text{dist}_{W}(t,A)=\inf_{\tilde{t}\in A}||t-\tilde{t}||_{W}$ for each $t$, where $||t||_{W}^{2}=t^{\prime}Wt$ for some symmetric, positive-definite matrix $W$.

Our implementation is based on the ‘stick-breaking’ parametrization of a categorical distribution from Linderman, Johnson and Adams (2015). Define the stick-breaking parametrization of the PMF $p$ as follows,

where $q_{k}:\mathcal{X}\rightarrow[0,1]$ for each $k=1,...,K-1$.¹⁴¹⁴ 14 For any discrete distribution $p$, there exists a well-defined stick-breaking representation because $q_{k}=p_{k}/\sum_{j\geq k}p_{j}$ for all $k=1,...,K-1$, so this parametrization does not entail any loss of generality. By substituting the stick-breaking weights, the likelihood function

satisfies $L_{n}(p)=L_{n}^{*}(q)$, where¹⁵¹⁵ 15 $L_{n}^{*}(q)$ presumes the support of $Y$ has been enumerated (with $y_{k}$ being an enumerated support point) so that the event $Y>y_{k}$ has meaning.

Consequently, by specifying a prior for $q:=(q_{1},...,q_{K-1})$ and computing the posterior for $(q(x_{1}),...,q(x_{n}))$ using $L_{n}^{*}(q)$, Bayesian inference for $(p(x_{1}),...,p(x_{n}))$ is obtained via the induced probability distribution for $p$ under the stick-breaking parametrization (1).

Since the stick-breaking functions $q$ only need to satisfy $0\leq q_{k}\leq 1$ for each $k=1,...,K-1$, we specify the prior for $q$ as the probability law of $(\Lambda(B_{1}),...,\Lambda(B_{K-1}))$, where $\Lambda(z)=\exp(z)/(1+\exp(z))$ for each $z\in\mathbb{R}$, and

with $GP(\mu_{k},\kappa_{k})$ denoting a Gaussian process with mean function $\mu_{k}:\mathcal{X}\rightarrow\mathbb{R}$ and covariance function $\kappa_{k}:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$.¹⁶¹⁶ 16 The parametrization $\Lambda(B_{k})$ is fully nonparametric because $B_{k}=\Lambda^{-1}(q_{k})$. Following similar constructions in the conditional density estimation literature (e.g., Ren et al. (2011)), we call this the logistic stick-breaking Gaussian process prior, and use $LSBGP(\{\mu_{k}\}_{k=1}^{K-1},\{\kappa_{k}\}_{k=1}^{K-1})$ to denote this class of priors. The specification of $\kappa_{k}$ allows for flexible incorporation of nonparametric properties like smoothness, while the choice of $\mu_{k}$ enables shrinkage towards a particular parametric model for $q$, such as a linear index model (Williams and Rasmussen, 2006). Moreover, the prior is compatible with our general assumptions. Indeed, under the GP, the vector $(B_{n,1},...,B_{n,K-1})$, with $B_{n,k}:=(B_{k}(x_{i}))_{i=1}^{n}$, satisfies

where $\mu_{n,k}:=(\mu_{k}(x_{i}))_{i=1}^{n}$ and $\kappa_{n,k}:=(\kappa_{k}(x_{i},x_{j}))_{i,j=1}^{n}$ for $k=1,...,K-1$. This implies that Assumption 2.1 holds because $B\mapsto p$ has a differentiable inverse. In practice, we set $\mu_{k}=0$ for each $k=1,...,K-1$, so prior elicitation amounts to specifying $\{\kappa_{k}\}_{k=1}^{K-1}$.

We describe sampling from the posterior for $(B_{n,1},...,B_{n,K-1})$, the quantities relevant for computing $(p(x_{1}),...,p(x_{n}))$. Under the independent GP prior, the posterior density for $(B_{n,1},...,B_{n,K-1})$ satisifes

where $\pi_{n,k}$ is the density of $\mathcal{N}(\mu_{n,k},\kappa_{n,k})$, $\mathcal{I}_{k}=\{i:Y_{i}\geq y_{k}\}$, and