Beta Survival Models
Extracted source text ยท arXiv 1905.03818
%
% The first command in your LaTeX source must be the \documentclass command.
%\documentclass[sigconf]{acmart}
%\documentclass[sigconf,anonymous,review]{acmart}
\documentclass[sigconf]{acmart}
% comment these 3 to get back to KDD format
\settopmatter{printacmref=false} % Removes citation information below abstract
\renewcommand\footnotetextcopyrightpermission[1]{} % removes footnote with conference information in first column
\pagestyle{plain} % removes running headers
\usepackage{listings}
% uncomment these to get back to NEFTLIX Black background
%\definecolor{nflx_black}{RGB}{34,30,30}
%\pagecolor{nflx_black}
%
% defining the \BibTeX command - from Oren Patashnik's original BibTeX documentation.
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08emT\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
% Rights management information.
% This information is sent to you when you complete the rights form.
% These commands have SAMPLE values in them; it is your responsibility as an author to replace
% the commands and values with those provided to you when you complete the rights form.
%
% These commands are for a PROCEEDINGS abstract or paper.
\copyrightyear{2019}
\acmYear{2019}
\setcopyright{acmlicensed}
\acmConference[KDD '19]{KDD '19: Symposium on foo}{August 04--08, 2019}{Anchorage, AK}
%\acmBooktitle{Woodstock '18: ACM Symposium on Neural Gaze Detection, August 04--08, 2019, Anchorage, AK}
%\acmPrice{15.00}
%\acmDOI{10.1145/1122445.1122456}
%\acmISBN{978-1-4503-9999-9/18/06}
%
% These commands are for a JOURNAL article.
%\setcopyright{acmcopyright}
%\acmJournal{TOG}
%\acmYear{2018}\acmVolume{37}\acmNumber{4}\acmArticle{111}\acmMonth{8}
%\acmDOI{10.1145/1122445.1122456}
%
% Submission ID.
% Use this when submitting an article to a sponsored event. You'll receive a unique submission ID from the organizers
% of the event, and this ID should be used as the parameter to this command.
%\acmSubmissionID{123-A56-BU3}
%
% The majority of ACM publications use numbered citations and references. If you are preparing content for an event
% sponsored by ACM SIGGRAPH, you must use the "author year" style of citations and references. Uncommenting
% the next command will enable that style.
%\citestyle{acmauthoryear}
%
% end of the preamble, start of the body of the document source.
\begin{document}
%\color{white} % uncomment to go back to nflx_black
%
% The "title" command has an optional parameter, allowing the author to define a "short title" to be used in page headers.
\title{Beta Survival Models}
%
% The "author" command and its associated commands are used to define the authors and their affiliations.
% Of note is the shared affiliation of the first two authors, and the "authornote" and "authornotemark" commands
% used to denote shared contribution to the research.
\author{David Hubbard, Beno{\^\i}t Rostykus, Yves Raimond, Tony Jebara}
\email{{dhubbard,brostykus,tjebara,yraimond}@netflix.com}
\affiliation{%
\institution{Netflix}
\streetaddress{100 Albright Way}
\city{Los Gatos}
\state{CA}
\postcode{95032}
}
%\author{David Hubbard}
%\authornotemark[1]
%\email{dhubbard@netflix.com,}
%\affiliation{%
% \institution{Netflix}
% \streetaddress{100 Albright Way}
% \city{Los Gatos}
% \state{CA}
% \postcode{95032}
%}
%\author{Beno{\^\i}t Rostykus}
%\authornotemark[1]
%\email{brostykus@netflix.com}
%\affiliation{%
% \institution{Netflix}
% \streetaddress{100 Albright Way}
% \city{Los Gatos}
% \state{CA}
% \postcode{95032}
%}
%\author{Yves Raimond}
%\email{yraimond@netflix.com}
%\affiliation{%
% \institution{Netflix}
%% \streetaddress{100 Albright Way}
% \city{Los Gatos}
% \state{CA}
% \postcode{95032}
%}
%\author{Tony Jebara}
%\authornote{all authors contributed equally to this research.}
%\email{tjebara@netflix.com}
%\orcid{1234-5678-9012}
%\affiliation{%
% \institution{Netflix}
% \streetaddress{100 Albright Way}
% \city{Los Gatos}
% \state{CA}
% \postcode{95032}
%}
%
% By default, the full list of authors will be used in the page headers. Often, this list is too long, and will overlap
% other information printed in the page headers. This command allows the author to define a more concise list
% of authors' names for this purpose.
\renewcommand{\shortauthors}{Hubbard, Rostykus, Raimond and Jebara.}
%
% The abstract is a short summary of the work to be presented in the article.
\begin{abstract}
This article analyzes the problem of estimating the time until an event occurs, also known as survival modeling. We observe through substantial experiments on large real-world datasets and use-cases that populations are largely heterogeneous. Sub-populations have different mean and variance in their survival rates requiring flexible models that capture heterogeneity. We leverage a classical extension of the logistic function into the survival setting to characterize unobserved heterogeneity using the beta distribution. This yields insights into the geometry of the problem as well as efficient estimation methods for linear, tree and neural network models that adjust the beta distribution based on observed covariates. We also show that the additional information captured by the beta distribution leads to interesting ranking implications as we determine who is most-at-risk. We show theoretically that the ranking is variable as we forecast forward in time and prove that pairwise comparisons of survival remain transitive. Empirical results using large-scale datasets across two use-cases (online conversions and retention modeling), demonstrate the competitiveness of the method. The simplicity of the method and its ability to capture skew in the data makes it a viable alternative to standard techniques particularly when we are interested in the time to event and when the underlying probabilities are heterogeneous.
\end{abstract}
%
% The code below is generated by the tool at http://dl.acm.org/ccs.cfm.
% Please copy and paste the code instead of the example below.
%
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10002950.10003648.10003688.10003694</concept_id>
<concept_desc>Mathematics of computing~Survival analysis</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257</concept_id>
<concept_desc>Computing methodologies~Machine learning</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257.10010258.10010259.10003268</concept_id>
<concept_desc>Computing methodologies~Ranking</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257.10010293.10003660</concept_id>
<concept_desc>Computing methodologies~Classification and regression trees</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010341.10010342</concept_id>
<concept_desc>Computing methodologies~Model development and analysis</concept_desc>
<concept_significance>300</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}
\ccsdesc[500]{Mathematics of computing~Survival analysis}
\ccsdesc[500]{Computing methodologies~Machine learning}
\ccsdesc[300]{Computing methodologies~Ranking}
\ccsdesc[300]{Computing methodologies~Classification and regression trees}
\ccsdesc[300]{Computing methodologies~Model development and analysis}
%
% Keywords. The author(s) should pick words that accurately describe the work being
% presented. Separate the keywords with commas.
\keywords{beta distribution, survival regression, ranking, nonlinear, boosting, heterogeneous}
%
% A "teaser" image appears between the author and affiliation information and the body
% of the document, and typically spans the page.
%\begin{teaserfigure}
% \includegraphics[width=\textwidth]{top_sim_surv.pdf}
% \caption{Heterogeneity gives rise to different survival distributions %and rankings.}
% \Description{showing the way different densities evolve over time, and %how in the end we can improve our rankings as a result of learning %this information.}
% \label{fig:teaser}
%\end{teaserfigure}
%
% This command processes the author and affiliation and title information and builds
% the first part of the formatted document.
\maketitle
\section{Introduction}
Survival modeling, customer lifetime value \cite{gupta2006} and product ranking \cite{RudinEtAl12,ChangEtAl2012} are of practical interest when we want to estimate time until a specific event occurs or rank items to estimate which will encounter the event first. Traditionally leveraged in medical applications, today survival regression is extensively used in large-scale business settings such as predicting time to conversion in online advertising and predicting retention (or churn) in subscription services. Standard survival regression involves a maximum likelihood estimation problem over a specified continuous distribution of the time until event (exponential for the Accelerated Failure Time model \cite{kalbfleisch2011statistical}) or of the hazard function (in the case of Cox Proportional Hazards \cite{cox1972regression}). In practice, time to event problems are often converted to classification problems by choosing a fixed time horizon which is appropriate for the application at hand. One then has to balance training the model on recent data against a longer labeling horizon which might be more desirable. Survival models avoid this trade-off by relying on right-censoring. This maps it to missing data problem where not all events are observed due to the horizon over which the data is collected.
There is evidence (which will be further demonstrated in this article) of the importance of heterogeneity in a variety of real-world time to event datasets. Heterogeneity indicates that items in a data-set have different survival means and variances. For instance, heterogeneity in a retention modeling context would be that as time increases the customers with the highest probability to retain are the ones which still remain in the dataset. Without considering this effect, it might appear that the baseline retention probability has increased over time when in fact the first order effect is that there is a mover/stayer bias. Thus methods which don't consider multiple decision points can fail to adequately account for this effect and thus fall victim to the so called ruse of heterogeneity \cite{vaupel1985}.
Consider the following example inspired by Porath \cite{ben1973labor} where we have 2 groups of customers, one in which the customers have a retention probability of $0.5$ and in the other the customers are uniformly split between retention probabilities of either $1.0$ or $0.0$. In this case after having observed only one decision point we would observe the retention probabilities of the two groups to be identical. However, if we consider multiple decision points it becomes clear that the latter population has a much higher long term retention rate because some customers therein retain to infinity. In order to capture this unobserved heterogeneity we need a distribution that is flexible enough to capture these dynamics and ideally as simple as possible. To that end we posit a beta prior on our instantaneous event probabilities. The beta has only 2 parameters, yet is flexible enough to capture right/left skewed, U-shaped, or normal distributions.
\begin{figure}
\includegraphics[width=\linewidth]{top_sim_surv_small.pdf}
\caption{Heterogeneity gives rise to different survival distributions and rankings.}
\Description{showing the way different densities evolve over time, and how in the end we can improve our rankings as a result of learning this information.}
\label{fig:teaser_small}
\end{figure}
Consider the example in Figure~\ref{fig:teaser_small}. A data-set contains three heterogeneous items (green dots, orange plus and blue cross). These items are each characterized by beta distributions (left panel). At each time period, each item samples a Bernoulli distributed coin from its beta distribution and flips it to determine if the item will retain. In the middle panel, we see the retention of the items over time and in the right-mode panel we see the ranking of the items over time. Even though the items are sampling from fixed beta distributions, the ranking of which item is most at risk over time changes. Thus, a stationary set of beta distributions lead to non-stationary survival and rankings. Such nuance cannot be captured by summarizing each item with only a point-estimate of survival (as opposed to a 2-parameter beta distribution).
Due to the discrete and repeat nature of the decision problems over time, we leverage a geometric hypothesis to recover survival distributions. We estimate the parameters of this model via an empirical Bayes method which can be efficiently implemented through the use of analytical solutions to the underlying integrals. This model termed the beta-logistic was first introduced by Heckman and Willis \cite{Heckman1977}, and was also studied by Fader and Hardie \cite{fader2007project}. We find that in practice this model fits the discrete decision data quite well, and that it allows for accurate projections of future decision points.
We extend the beta-logistic model to the case of large-scale trees or neural-network models that adjust the beta distribution given input covariates. These leverage the use of recurrence relationships to efficiently compute the gradient. Through the beta prior underpinning the model, we show empirically that the beta-logistic is able to model a wide range of heterogeneous behaviors that other survival or binary classification models fail to capture, especially in the presence of skew. As we will see, the beta-logistic model outperforms a typical binary logistic regression in real-world examples, and provides tighter estimated posteriors compared to a typical Laplace approximation.
We also present theoretical results on ranking with beta distributions. We show that pairwise comparisons between beta distributions can be summarized by the median of the two distributions. This makes ranking with beta distributions a provably transitive problem (pairwise distribution comparisons are generally non-transitive \cite{Savage1994}). Therefore, using the medians of beta distributions allows us to rank the entire population to see which subscribers or which items are most-at-risk. The results are then extended to the case where we rank items across multiple time horizons by approximating the evolution of the survival distribution over time as a product of beta distributions as in \cite{fan1991distribution}. Thus we obtain a consistent ranking of items which evolves over time and is summarized by medians (rather than means) to improve the accuracy of ranking who is most-at-risk.
This paper is organized as follows. We first show the beta-logistic derivation as well as reference the recursion formulas which make the computation efficient. We also make brief remarks about convexity and observe that in practice we rarely encounter convergence issues. We then present several simulated examples to help motivate the discussion. This is followed by an empirical performance evaluation of the various models across three large real-world datasets: a sparse online conversion dataset and two proprietary datasets from a popular video streaming service involving subscription and viewing behaviors. In all the examples, the beta-logistic outperforms other baseline methods, and seems to perform better in practice regardless of how many attributes are considered. Even though there will always be unobserved variations between individuals that influence the time-to-event, this empirical evidence is consistent.
\section{The Beta-Logistic for Survival Regression}
\subsection{Model derivation}
Denote by $(x_i, t_i, c_i) \in \mathbb{R}^d \times \mathbb{N}\times \{0, 1\}$ a dataset where $x_i$ are covariates, $t_i$ is the discrete time to event for an observed (i.e. uncensored) datapoint ($c_i = 0$) and $t_i$ is the right-censoring time for a datapoint for which the event of interest hasn't happened yet ($c_i = 1)$. A traditional survival model would posit a parametric distribution $p(T|x)$ and then try to maximize the following empirical likelihood over a class of functions $f$:
\begin{equation*}
L = \prod_{\forall i, c_i = 0} \mathbb{P}\left(T=t_i| f(x_i)\right)\prod_{\forall i, c_i = 1} \mathbb{P}\left(T > t_i | f(x_i)\right).
\end{equation*}
Unfortunately, unless constrained to a few popular distributions such as the exponential one, the maximization of such a quantity is usually intractable for most classes of functions $f(x)$.
Let us instead assume that at each discrete decision point, a customer decides to retain with some (point-estimate) probability $1 -\theta$ where $\theta$ is some function of the covariates $x$. Then we further assume that the instantaneous event probability at decision point $t$ is characterized by a shifted geometric distribution as follows:
\begin{equation*}
\mathbb{P}(T=t|\theta) = \theta (1-\theta)^{t-1}, ~
where ~ \theta \in \left[0,1\right].
\end{equation*}
This then gives the following survival equation:
\begin{equation}
\mathbb{P}(T > t|\theta) = 1-\sum_{i=1}^{t}\mathbb{P}(T=i|\theta).
\label{eqn:survival}
\end{equation}
This geometric assumption follows from the discrete nature of the decisions customers need to make in a subscription service, when continuing to next episodes of a show, etc. It admits a a simple and straightforward survival estimate that we can also use to project beyond our observed time horizon. Now in order to capture the heterogeneity in the data, we can instead assume that $\theta$ follows a conditional beta prior $(\mathbb{B})$ as opposed to being a point-estimate as follows:
\begin{equation*}
f(\theta|\alpha(x),\beta(x))=\frac{\theta^{\alpha(x)-1}(1-\theta)^{\beta(x)-1}}{B(\alpha(x),\beta(x))}
\end{equation*}
where $\alpha(x)$ and $\beta(x)$ are some arbitrary positive functions of covariates (e.g. measurements that characterize a specific customer or a specific item within the population).
Consider the Empirical Bayes method \cite{gelman2013bayesian} (also called Type-II Maximum Likelihood Estimation \cite{berger2013statistical}) as an estimator for $\alpha(x)$ and $\beta(x)$ given the data:
\begin{equation*}
\max_{\alpha, \beta} L(\alpha, \beta)
\end{equation*}
where
\begin{equation}\label{eq:likelihood_type2}
L(\alpha, \beta) = \prod_{\forall i,c_i = 0} \mathbb{P}\left(T=t_i| \alpha(x_i),\beta(x_i)\right)\prod_{\forall i, c_i = 1} \mathbb{P}\left(T > t_i | \alpha(x_i),\beta(x_i)\right).
\end{equation}
Using the marginal likelihood function we obtain:
\begin{equation*}
\mathbb{P}(T | \alpha(x), \beta(x)) = \int_{0}^{1}f(\theta|\alpha(x),\beta(x))\mathbb{P}(T | \theta)d\theta.
\end{equation*}
As we will see in the next section, a key property of the beta-logistic model is that it makes the maximization of Equation~\ref{eq:likelihood_type2} tractable.
Since $\alpha$ and $\beta$ have to be positive to define valid beta-distributions, we use an exponential reparameterization and aim to estimate functions $a(x)$ and $b(x)$ such that:
\begin{equation*}
\alpha(x) = e^{a(x)} ~ and ~ \beta(x) = e^{b(x)}.
\end{equation*}
Throughout the paper, we will also assume that $a$ and $b$ are twice-differentiable.
The name \textbf{beta-logistic} for such a model has been coined by \cite{Heckman1977} and studied when the predictors $a(x) = \gamma_a \cdot x$ and $b(x) = \gamma_b \cdot x$ are linear functions. In this case, at $T=1$ observe that if we want to estimate the mean this reduces to an over-parameterized logistic regression:
\begin{equation*}
\mathbb{P}(T = 1 | \alpha(x), \beta(x)) = \frac{\alpha(x)}{\alpha(x)+\beta(x)} =\frac{1}{1+e^{(\gamma_b-\gamma_a)^\top x}}.
\end{equation*}
\subsection{Algorithm}
We will now consider the general case where $a(x)$ and $b(x)$ are nonlinear functions and could be defined by the last layer of a neural network. Alternatively, they could be generated by a vectored-output Gradient Boosted Regression Tree (GBRT) model.
Using properties of the beta function (see \ref{subsection:recurrence_derivation}), one can show that:
\begin{equation*}\label{eq:t_1_p_formula}
\mathbb{P}(T = 1 | \alpha, \beta) = \frac{\alpha}{\alpha+\beta}
\end{equation*}
and
\begin{equation*}\label{eq:t_1_s_formula}
\mathbb{P}(T > 1 | \alpha, \beta) = \frac{\beta}{\alpha+\beta}.
\end{equation*}
Further, the following recursion formulas hold for $t > 1$:
\begin{equation}\label{eq:recursion_prod_p}
\mathbb{P}(T = t | \alpha, \beta) = \left( \frac{\beta+t-2}{\alpha+\beta+t-1} \right)\mathbb{P}(T = t - 1 | \alpha, \beta)
\end{equation}
and
\begin{equation}\label{eq:recursion_prod_s}
\mathbb{P}(T > t | \alpha, \beta) = \left( \frac{\beta+t-1}{\alpha+\beta+t-1} \right)\mathbb{P}(T > t - 1 | \alpha, \beta).
\end{equation}
If we denote $\ell = -\log L$ as the function we wish to minimize, Equation~\ref{eq:recursion_prod_p} and Equation~\ref{eq:recursion_prod_s} allow us to derive (see Appendix~\ref{subsection:appendix_gradient}) recurrence relationships for individual terms of $\dfrac{\partial \ell}{\partial \cdot}$ and $\dfrac{\partial^2 \ell}{\partial \cdot^2}$. This makes it possible for example to implement a custom loss gradient and Hessian callbacks in popular GBRT libraries such as XGBoost \cite{chen2016xgboost} and lightGBM \cite{ke2017lightgbm}. In this case, the GBRT models have "vector-output" and predict for every row both $a = \log(\alpha)$ and $b = \log(\beta)$ jointly from a set of covariates, similarly to how the multinomial logit loss is implemented in these libraries. More precisely, choosing a squared loss for the split criterion in each node as is customary, the model will equally weight how well the boosting residuals (gradients) with respect to $a$ and $b$ are regressed on.
Note that because of the inherent discrete nature of the beta-logistic model, the computational complexity of evaluating its gradient on a given datapoint is proportional to the average value of $t_i$. Therefore, a reasonable time step discretization value needs to be chosen to properly capture the survival dynamics while allowing fast inference. One can similarly implement this loss in deep learning frameworks. One would typically explicitly pad the label vectors $t_i$ with zeros up to the max censoring horizon (which would bring average computation per row to $O(\max_i t_i)$ for the mini-batch) so that computation can be expressed through vectorized operations, or via frameworks such as Vectorflow \cite{rostykusvectorflow} or Tensorflow \cite{abadi2016tensorflow} ragged tensors that allow for variable-length inputs to bring the computational cost back to $O(\text{avg}_i t_i)$.
\subsection{Convexity}
For brevity, define $\alpha_i = \alpha(x_i)$ and $\beta_i = \beta(x_i)$. In the special case where $a(x) = \gamma_{a}\cdot x$ and $b(x) = \gamma_{b}\cdot x$ are linear functions, their second derivative is null and the Hessian of the log-likelihood Equation~\ref{eq:likelihood_type2} is diagonal:
\begin{equation*}\label{eq:convexity_hess_a}
\frac{\partial^2 \ell}{\partial \gamma_{a,j}^2} = \gamma_{a,j}^2\alpha_i\sum_{i}\left[ \frac{\beta_i}{\left(\alpha_i + \beta_i\right)^2} + \sum_{u=2}^{t_i}\frac{\beta_i + u - 1}{\left(\alpha_i + \beta_i +u-1\right)^2}\right]
\end{equation*}
and
\begin{equation}\label{eq:convexity_hess_b}
\begin{aligned}
\frac{\partial^2 \ell}{\partial \gamma_{b,j}^2} = \gamma_{b,j}^2\beta_i\sum_{i}\left[ \frac{\alpha_i}{\left(\alpha_i + \beta_i\right)^2} + \sum_{u=2}^{t_i}k_i\right]
\end{aligned}
\end{equation}
where
\begin{equation*}
k_i =
\begin{cases}
(\alpha_i+1)\frac{\beta_i^2 - (u-2)(\alpha_i + u - 1)}{\left(\beta_i+u-2\right)^2\left(\alpha_i+\beta_i+u-1\right)^2} & \text{if $c_i = 0$} \\
\alpha_i\frac{\beta_i^2 - (u-1)(\alpha_i + u-1) }{\left(\beta_i+u-1\right)^2\left(\alpha_i+\beta_i+u-1\right)^2} & \text{otherwise.}
\end{cases}
\end{equation*}
%\begin{equation}\label{eq:convexity_hess_b_old}
% \begin{aligned}
% \frac{\partial^2 \ell}{\partial \gamma_{b,j}^2} = & \gamma_{b,j}^2\sum_{c_j=0}\left[ \frac{\alpha_i\beta_i}{\left(\alpha_i + \beta_i\right)^2} \\ & + \sum_{u=2}^{t_i}(\alpha_i+1)\beta_i\frac{\beta_i^2 - (u-2)(\alpha_i + u - 1)}{\left(\beta_i+u-2\right)^2\left(\alpha_i+\beta_i+u-1\right)^2}\right] \\
% & +\gamma_{b,j}^2\sum_{c_i=1}\left[ \frac{\alpha_i\beta_i}{\left(\alpha_i + \beta_i\right)^2} \\
% & + \sum_{u=2}^{t_i}\alpha_i\beta_i\frac{\beta_i^2 - (u-1)(\alpha_i + u-1) }{\left(\beta_i+u-1\right)^2\left(\alpha_i+\beta_i+u-1\right)^2}\right]
% \end{aligned}
%\end{equation}
We see that the log-likelihood of the shifted-beta geometric model is always convex in $\alpha$ when $a$ is linear. Further we can see that when all points are observed (no censoring), and the maximum horizon is $T=2$ then Equation~\ref{eq:convexity_hess_b} is also convex in $b$.
Subsequent terms are not convex, however, but despite that in practice we do not encounter significant convexity issues (e.g. local minima and saddle points). It seems likely that in practice the convex terms of the likelihood dominate the non-convex terms. Note once again that there is generally no global convexity of the objective function.
%\subsubsection{Non-linear Case}
%When T = 1, we have the following:\\
%\begin{theorem}
%If there exists constants $\omega_1 > 0$ and $\omega_2 > 0$ such that $\forall x$:
%\begin{equation}\label{eq:bound_skew_cond_convexity}
% \omega_1 \leq \dfrac{\alpha(x)}{\beta(x)} \leq \omega_2
%\end{equation}
%\begin{equation}
%\begin{aligned}
% 0 \leq \dfrac{\partial^2 a}{\partial \gamma_{a,i}^2} \leq \dfrac{K_a^2}{1 + \omega_1^{-1}}\\
% 0 \leq \dfrac{\partial^2 b}{\partial \gamma_{b,i}^2} \leq \dfrac{K_b^2}{1 + \omega_2}
%\end{aligned}
%\end{equation}
%Then the problem is globally convex in $a$ and $b$ for timestep $T=1$.
%\end{theorem}
%The condition (\ref{eq:bound_skew_cond_convexity}) essentially imposes a maximum right and left skew on the beta fitted.
%\begin{proof}
%At timestep 1, one can write the diagonal of the Hessian of the uncensored loss for a given data point as:
%\begin{equation}\label{eq:convexity_non_linear_hess_a}
%\begin{aligned}
% \frac{\partial^2 \ell_u}{\partial \gamma_{a,i}^2} = \frac{\partial^2 a}{\partial \gamma_{a,i}^2}\left(\dfrac{\alpha}{\alpha+\beta} - 1 \right) + \left(\frac{\partial %a}{\partial \gamma_{a,i}} \right)^2\dfrac{\alpha\beta}{\left(\alpha + \beta \right)^2}\\
% \frac{\partial^2 \ell_u}{\partial \gamma_{a,i}^2} = \frac{\partial^2 a}{\partial \gamma_{a,i}^2}\left(\dfrac{\alpha}{\alpha+1} - 1 \right) + \left(\frac{\partial a}{\partial %\gamma_{a,i}} \right)^2\dfrac{\alpha\beta}{\left(\alpha + \beta \right)^2}
%\end{aligned}
%\end{equation}
%\end{proof}
% Probably not include this part:
%For subsequent timesteps, we have the following:\\
%\begin{theorem}
%If $a(x)$ is a twice differentiable convex function (not-necessarily linear) and if:
%\begin{equation}
% \forall x_j s.t c_j = 0, t_j > 1: \alpha_j\log\left( 1 + \dfrac{t_j}{\alpha_j+\beta_j}\right) \geq 1
%\end{equation}
%Then the problem is convex in $a$.
%\end{theorem}
%\begin{corollary}
%If $a(x)$ is convex and if for all uncensored datapoints with $t_j > 1$:
%\begin{equation}
% \alpha_j \geq 1 ~ and ~ \alpha_j - 1 \geq \beta_j
%\end{equation}
%Then the problem is convex in $a$.
%\end{corollary}
%\begin{proof}
%This derives easily from the fact that $\forall x \geq 0$ $$\log(1+x) > \dfrac{2x}{2 + x}$$
%\end{proof}
\section{Ranking with the Beta-Logistic}
Given $n$ beta distributions another relevant question for the business is how do we rank them from best to worst? This is crucial if, for instance, products need to be ranked to prioritize maintenance as in \cite{RudinEtAl12}. For example, we may be interested in ranking beta distributions for news articles where we are estimating the probability that an article will be clicked on. Or for a video subscription service we could have $n$ beta distributions over $n$ titles each of which represents the probability that this title will be watched to the final episode (title survival). How do we rank the $n$ titles (articles) from most watchable (readable) to least watchable? Let us abstractly view both problems as interchangeable where we are ranking items.
We have two items ($u$ and $v$), each with their own $\alpha,\beta$ parameters that define a beta distribution. Each beta distribution samples a coin with a probability $1-\theta$ of heads (retaining) and $\theta$ for tails (churning out). In the first time step, item $v$ retains less than $u$ if it has a higher coin flip probability, e.g. $p(\theta_v > \theta_u)>0.5 $ or the probability is larger than 50\%. In the case of integer parameters, this is given by the following integral:
\begin{equation*}
\begin{split}
%\begin{align*}
p(\theta_v > \theta_u) & = \int_{\theta_u=0}^1 \int_{\theta_v=\theta_u}^1 \frac{1}{B(\alpha_u,\alpha_v)} \theta_u^{\alpha_u-1} (1-\theta_u)^{\beta_u-1}\cdot \\ & \frac{1}{B(\alpha_v,\alpha_v)} \theta_v^{\alpha_v-1} (1-\theta_v)^{\beta_v-1} d\theta_u d\theta_v
%\end{align*}
\end{split}
\end{equation*}
which simplifies \cite{miller_2015} into
\begin{equation}\label{eq:ranking_proba_formula}
p(\theta_v > \theta_u) = \sum_{i=0}^{\alpha_v-1} \frac{B(\alpha_u+i,\beta_u+\beta_v) } { (\beta_v + i) B(1+i,\beta_v)B(\alpha_u,\beta_u) }.
\end{equation}
If the quantity is larger than 0.5, then item $u$ retains less than item $v$ in the first time step. It turns out, under thousands of simulations, the less likely survivor is also the one with the larger median $\theta$. Figure~\ref{fig:medianVprobability} shows a scatter plot as we randomly compare pairs of beta distributions. It is easy to see that the difference in their medians agrees with the probability $p(\theta_v > \theta_u)$. So, instead of using a complicated formula to test $p(\theta_v > \theta_u)$, we just need to compare the medians via the inverse incomplete beta function (\texttt{betaincinv}) denoted by $I^{-1}()$ and see if $I^{-1}(0.5,\alpha_u,\beta_u) > I^{-1}(0.5,\alpha_v,\beta_v)$. A proof of this is below.
\begin{figure}
\includegraphics[width=\linewidth]{medianVprobability.pdf}
\caption{The medians of beta distributions are consistent with the pairwise probabilistic ranking of beta distributions.}
\Description{medians are consistent with pairwise ranking.}
\label{fig:medianVprobability}
\end{figure}
For later time steps, we will leverage a geometric assumption but applied to distributions rather than point estimates. The item which retains longer is the one with the lower product of repeated coin flip probabilities, i.e. $p(\theta_v^t > \theta_u^t)$. In that case, the beta distributions get modified by taking them to powers. Sometimes, the product of beta-distributed random variables is beta-distributed \cite{fan1991distribution} but other times it is {\em almost} beta distributed. In general, we can easily compute the mean and variance of this power-beta distribution to find a reasonable beta approximation to it with its own $\alpha,\beta$ parameters. This is done by leveraging the derivation of \cite{fan1991distribution} as follows. Assume we have a beta distribution $p(\theta|\alpha,\beta)$ and define the new random variable $z=(1-\theta)*\theta^{t-1}$. We can then derive the first moment as
\begin{equation*}
S=E_{p(z)}[z] = \left ( \frac{\beta}{\alpha+\beta} \right )\left ( \frac{\alpha}{\alpha+\beta} \right )^{t-1}
\end{equation*}
and the second moment as
\begin{equation*}
T=E_{p(z)}[z^2] = \left(\frac{\alpha(\alpha+1)}{(\alpha+\beta)(\alpha+\beta+1)} \right )\left(\frac{\beta(\beta+1)}{(\alpha+\beta)(\alpha+\beta+1)} \right )^{t-1}.
\end{equation*}
Then, we approximate the distribution for the future event probabilities $p(z)$ by a beta distribution where
\begin{equation}
\begin{aligned}
\hat{\alpha} & = \frac{(S-T)S}{T-S^2} \\
\hat{\beta} & = \frac{(S-T)(1-S)}{(T-S^2)}.
\label{eqn:evolution_of_beta_approximation}
\end{aligned}
\end{equation}
Therefore, in order to compare who is more likely to survive in future horizons, we can combine Equation~\ref{eqn:evolution_of_beta_approximation} and Equation~\ref{eqn:survival} to find the median of the approximated future survival distribution.
\begin{theorem}
\label{thm:maintheorem}
For random variables $\theta_u \sim {\rm Beta}(\alpha_u,\beta_u)$ and $\theta_v \sim {\rm Beta}(\alpha_v,\beta_v)$ for $\alpha_u,\alpha_v,\beta_u,\beta_v \in \mathbb{N}$,
$p(\theta_v > \theta_u) > 0.5$ if and only if $I^{-1}(0.5,\alpha_v,\beta_v) > I^{-1}(0.5,\alpha_u,\beta_u)$ (the median of $\theta_v$ is larger than the median of $\theta_u$).
\end{theorem}
\begin{proof}
We first prove that the median gives the correct winner under simplifying assumptions when both beta distributions have the same $\alpha$ or the same $\beta$.
First consider when the distributions have the same $\alpha_u=\alpha_v=\alpha$ and different $\beta_u$ and $\beta_v$. In that case, Equation~\ref{eq:ranking_proba_formula} simplifies to
\begin{equation}
p(\theta_v > \theta_u) = \sum_{i=0}^{\alpha_v-1} \frac{ B(\alpha_u+i,\beta_u+\beta_v) } { (\beta_v + i) B(1+i,\beta_v) B(\alpha,\beta_u) }.
\end{equation}
The formulas for $p(\theta_v > \theta_u)$ and $p(\theta_u > \theta_v) $ only differ in their denominators. Then, if $\beta_v > \beta_u$ it is easy to show that
\begin{equation}
(\beta_v + i) B(1+i,\beta_v) B(\alpha,\beta_u) > (\beta_u + i) B(1+i,\beta_u) B(\alpha,\beta_v).
\end{equation}
Therefore, $p(\theta_v > \theta_u) > p(\theta_u > \theta_v)$ if and only if $\beta_v > \beta_u$.
Similarly, if $\beta_v > \beta_u$, the medians satisfy $I^{-1}(0.5,\alpha_v,\beta_v) < I^{-1}(0.5,\alpha_u,\beta_u)$. This is true since, if all else is equal, increasing the $\beta$ parameter reduces the median of a beta distribution. Therefore, for $\alpha_u=\alpha_v$, the median ordering is always consistent with the probability test.
An analogous derivation holds when the two distributions have the same $\beta_u=\beta_v$ and different $\alpha_u$ and $\alpha_v$. This is obtained by using the property $p(\theta|\alpha,\beta)=p(1-\theta|\beta,\alpha)$. Therefore, for $\beta_u=\beta_v$, the median ordering is always consistent with the probability test.
Next we generalize these two statements to show that the median ordering always agrees with the probability test. Consider the situation where $median(\alpha_u,\beta_u)>median(\alpha_u,\beta_v)>median(\alpha_v,\beta_v)$. Due to the scalar nature of the median of the beta distribution, we must have transitivity. We must also have $median(\alpha_u,\beta_u) < median(\alpha_v,\beta_u) < median(\alpha_v,\beta_v)$. Since each pair of inequalities on medians requires that the corresponding statement on the probability tests also holds, the overall statement $median(\alpha_u,\beta_u)<median(\alpha_v,\beta_v)$ must also imply that $p(\theta_v > \theta_u) > p(\theta_u > \theta_v)$.
\end{proof}
Therefore, thanks to Theorem~\ref{thm:maintheorem}, we can safely rank order beta distributions simply by considering their medians. These rankings are not only pairwise consistent but globally consistent. Recall that pairwise ranking of distributions does not always yield globally consistent rankings as popularly highlighted through the study of nontransitive dice \cite{Savage1994}. Thus, given any beta distribution at a particular horizon, it is straightforward to determine which items are most at risk through a simple sorting procedure on the medians. Given this approach to ranking beta distributions, we can show the performance of our model-based ranking of users or items by holding out data and evaluating the time to event in terms of the AUC (area under the curve) of the receiver operating characteristic (ROC) curve.
\section{Empirical Results}
\subsection{Synthetic simulations}
We present simulation results for the beta-logistic, and compare them to the logistic model. We also show that the beta-logistic successfully recovers the posterior for skewed distributions. In our first simulation we have 3 beta distributions which have very different shapes (see table \ref{table:simulation1} below), but with the same mean (this example is inspired by Fader and Hardie \cite{fader2018project}). Here, each simulated customer draws a coin from one of these distributions, and then flips that coin repeatedly until they have an event or we reach a censoring horizon (in this particular case we considered 4 decision points).
\begin{table}[h]
%\color{white} % uncomment to get back to nflx_black
\begin{tabular}{ |p{2cm}||p{1cm}|p{1cm}|p{1cm}| }
\hline
shape & $\alpha$ & $\beta$ & $\mu$\\
\hline
normal & 4.75 & 14.25 & 0.25 \\
right skewed & 0.5 & 1.50 & 0.25 \\
u shaped & $0.08\bar{3}$ & 0.25 & 0.25 \\
\hline
\end{tabular}
\caption{Heterogeneous beta distributions with identical means.}
\label{table:simulation1}
\end{table}
%\begin{figure}[h]
% \centering
% \includegraphics[width=\linewidth]{simulation_1.pdf}
% \label{fig:simulation1}
% \Description{Simulation 1}
%\end{figure}
It is trivial to show that the logistic model will do no better than random in this case, because it is not observing the dynamics of the survival distribution which reveal the differing levels of heterogeneity underlying the 3 populations. If we allow the beta-logistic model to have a dummy variable for each of these cases then it can recover the posterior of each (see Figure~\ref{fig:simulation1_survival}). This illustrates an important property of the beta-logistic: it recovers posterior estimates even when the data is very heterogeneous and allows us to fit survival distributions well.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{simulation_1_surv.pdf}
\caption{Survival distributions as a function of time as well as an estimate of $\hat{S}(t)$ from the beta-logistic. Using a point-estimate of the mean $\Bar{\theta}$ (as in the logistic model) fails to recover the heterogeneity.}
\label{fig:simulation1_survival}
\Description{Simulation 1}
\end{figure}
To create a slightly more realistic simulation, we can include another term which increases the homogeneity linearly in $\alpha$ and we add this as another covariate in the models. We also inject exponential noise into the $\alpha$ and $\beta$ used for our random draws. Now, the logistic model does do better than random when there is homogeneity present (see Figure~\ref{fig:simulation2}), however it still leaves signal behind by not considering the survival distribution. We additionally show results for a one time step beta logistic which performs similarly to the logistic model. However it seems to have a lower variance which perhaps indicates that its posterior estimates are more conservative, a property which will be confirmed in the next set of experiments.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{simulation_2_w_homogeneity_no_legend.pdf}
\caption{The level of heterogeneity increases from the left panel to the right panel as we add a linear term in $\alpha$. Clearly, the mean of the beta-logistic 1 step (magenta plus), and logistic (cyan dot) are nearly identical, but the beta-logistic (orange cross) considers more survival information and outperforms both even when there is considerable homogeneity.}
\Description{Simulation 2}
\label{fig:simulation2}
\end{figure}
\subsection{Online conversions dataset}
\subsubsection{Survival modeling}
We now evaluate the performance of the beta logistic model on a large-scale sparse dataset. We use the Criteo online conversions dataset published alongside \cite{chapelle2014modeling} and publicly available for download\footnote{http://labs.criteo.com/2013/12/conversion-logs-dataset/}. We consider the problem of modeling the distribution of the time between a click event and a conversion event. We will consider a censoring window of 12 hours (61\% of conversions happen within that window). As noted in \cite{chapelle2014modeling}, the exponential distribution fits reasonably well the data so we will compare the beta-logistic model against the exponential distribution (1 parameter) and the Weibull distribution (2 parameters). Since the temporal integration of the beta-logistic model is intrinsically discrete, we consider a time-discretization of 5 minute steps. We also add as baselines 2 logistic models: one trained at a horizon of 5 minutes (the shortest interval), and one trained at a horizon of 12 hours (the largest window). All conditional models are implemented as sparse linear models in Vectorflow \cite{rostykusvectorflow} and trained through stochastic gradient descent. All survival models use an exponential reparameterization of their parameters (since the beta, exponential, and Weibull distributions all require positivity in their parameters). Censored events are down-sampled by a factor of 10x. We use 1M rows for training and 1M (held-out in time) rows for evaluation. The total (covariate) dimensionality of the problem is 102K after one-hot-encoding. Note that covariates are sparse and the overall sparsity of the problem is over 99.98\%. Results are presented in Figure~\ref{fig:criteo_exp1}.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{criteo_exp_1.png}
\caption{AUC as a function of censoring horizon for the various models considered.}
\label{fig:criteo_exp1}
\Description{Online conversion experiment 1}
\end{figure}
The beta-logistic survival model outperforms other baselines at all horizons considered.
Even though it is a 2-parameter distribution, the Weibull model is interestingly performing worse than the exponential survival model and the binary logistic classifier. We hypothesize that this is due to the poor conditioning of its loss function as well as the numerical instabilities during gradient and expectation computation (the latter requires function calls to the gamma function which is numerically difficult to estimate for moderately small values and for large values).
\subsubsection{Posterior size comparison}
We next consider the problem as a binary classification task (did a conversion happen within the specified time window?). It is interesting to compare the confidence interval sizes of various models. For the conditional beta-logistic model, the prediction variance on datapoint $x$ is given by:
\begin{equation*}\label{eq:beta_logistic_variance}
\text{Var}(x) = \dfrac{\alpha(x)\beta(x)}{\left(\alpha(x)+\beta(x)\right)^2\left(\alpha(x)+\beta(x)+1\right)}.
\end{equation*}
For a logistic model parameterized by $\theta\in\mathbb{R}^d$, a standard way to estimate the confidence of a prediction is through the Laplace approximation of its posterior \cite{mackay2003information}. In the high-dimensional setting, estimating the Hessian or its inverse become impractical tasks (storage cost is $O(d^2)$ and matrix inversion requires $O(d^3)$ computation). In this scenario, it is customary to assume independence of the posterior coordinates and restrict the estimation to the diagonal of the Hessian $h = \dfrac{1}{\sigma^2} \in \mathbf{R}^d$, which reduces both storage and computation costs to $O(d)$. Hence under this assumption, for a given datapoint $x$ the distribution of possible values for the random variable $Y = \theta^T x$ is also Gaussian with parameters:
\begin{equation*}
\mathcal{N}\left(\sum_i \theta_i x_i, \sum_i \sigma_{i}^2x_i^2 \right).
\end{equation*}
If the full Hessian inverse $H^{-1}$ is estimated, then $Y$ is Gaussian with parameters:
\begin{equation*}
\mathcal{N}\left(\theta \cdot x, x^T \cdot H^{-1}x)\right).
\end{equation*}
When $Y$ is Gaussian, the logistic model prediction
\begin{equation*}
\mathbb{P}(T=1 \vert x, \theta) = \dfrac{1}{1 + \exp(-Y)}
\end{equation*}
has a distribution for which the variance $v$ can be conveniently approximated. See \cite{li2012unbiased} for various suggested approximations schemes. We chose to apply the following approximation
\begin{equation*}
v = \Phi\left(\dfrac{\pi\mu/\sqrt{8} -1}{\sqrt{\pi -1 + \pi^2\sigma^2/8}} \right) - \left(1 + \exp(-\mu/\sqrt{1 + \pi\sigma^2/8}) \right)^{-2}.
\end{equation*}
Armed with this estimate for the logistic regression posterior variance, we run the following experiment: we random-project (using Gaussian vectors) the original high-dimensional data into a 50-dimensional space, in which we train beta-logistic classifiers and logistic classifiers at various horizons, using 50k training samples every time. We then compare the average posterior size variance on a held-out dataset containing 50k samples. Holdout AUCs were comparable in this case for both models at all horizons. Two posterior approximations are reported for the logistic model: one using the full Hessian inverse and the other one using only the diagonalized Hessian. Results are reported in Figure~\ref{fig:criteo_exp2}.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{criteo_exp_2.png}
\caption{The posterior variance of beta-logistic binary classifiers as well as logistic regressions trained on binary labels datasets with increasing censoring windows.}
\label{fig:criteo_exp2}
\Description{Online conversion experiment 2}
\end{figure}
Note that the beta-logistic model produces much smaller uncertainty estimates (between 20\% and 45\% smaller) than the logistic model with Laplace approximation. Furthermore, the growth rate as a function of the horizon of the binary classifier is also smaller for the beta-logistic approach. Also note that the Laplace posterior with diagonal Hessian approximation underestimates the posterior obtained using the full Hessian. Gaussian posteriors are obviously unable to appropriately model data skew.
This empirical result is arguably clear evidence of the superiority of the posteriors generated by the beta-logistic model over a standard Laplace approximation estimate layered onto a logistic regression model posterior. The beta-logistic posterior is also much cheaper to recover in terms of computational and storage costs.
This also suggests that the beta-logistic model could be a viable alternative to standard techniques of explore-exploit models in binary classification settings.
\subsection{Video streaming subscription dataset}
\subsubsection{Retention modeling}
We now study the problem of modeling customer retention for a subscription business. We leverage a proprietary dataset from a popular video streaming service. In a subscription setting when a customer chooses not to renew the service, the outcome is explicitly observed and logged. From a practical perspective, it is obviously preferable and meaningful when a customer's tenure with the service is n months rather than 1 month. It is also clearly valuable to be able to estimate and project that tenure accurately across different cohorts of customers. In this particular case the cohorts are highly heterogeneous as shown in Figure~\ref{fig:netflix_beta}.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{netflix_betas.pdf}
\caption{Estimated churn probabilities for 3 different cohorts. The large variations in the shape of the fitted distributions motivate the use of a beta prior on the conditional churn probability.}
\Description{Subscription Retention Model AUC}
\label{fig:netflix_beta}
\end{figure}
In this example the data set had more than 10M rows and 500 columns. We trained 20 models on bootstraps of the data with a 10x downsample on censored customers. We used 4 discrete decision points to fit the model. Evaluation was done on a subset of 3M rows which was held out from the models, and held out in time as well over an additional 5 decision points (9 total months of data). All models are implemented as GBRTs and estimated using lightGBM. In Figure~\ref{fig:sub_auc}, we show the evaluation of the models across two cohorts: one with relatively little data and covariates to describe the customers (which should clearly benefit from modeling the unobserved heterogeneity) and one with much richer data and covariates. Surprisingly even on the rich data set where one might argue there should be considerable homogeneity within any given subset of customers, we still find accuracy improvements by using the beta-logistic over the standard logistic model. This example illustrates how regardless of how many covariates are considered, there is still considerable heterogeneity.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{netflix_actual_both_auc.pdf}
\caption{Held-out AUC for two different cohorts of customers.}
\Description{AUC of the Subscription Retention Model.}
\label{fig:sub_auc}
\end{figure}
\subsubsection{Retention within shows}
Another problem of importance to a video subscription business is ranking shows that customers are most likely to fully enjoy (i.e. customers watch the shows to completion across all the episodes). Here we model the distribution of survival of watched episodes of a show conditional on the customer having started the show. % (6 minutes on the first episode).
In Figure~\ref{fig:netflix completion} we compare the performance of the beta-logistic to logistic models at an early (1 episode) horizon and a late horizon (8 episodes). The dataset contains 2k shows and spans 3M rows and 500 columns. We used a 50/50 train/test split. Model training used bootstrap methods to provide uncertainty estimates. All models are implemented as GBRTs which were estimated in lightGBM. In early horizons, the beta-logistic model once again provides significant AUC improvements over logistic models trained at either the 1 episode horizon and 8 episode horizon.
\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{netflix_completion_w_legend.pdf}
\caption{The beta-logistic improves ranking accuracy (in terms of AUC) for early horizons.}
\label{fig:netflix completion}
\end{figure}
\section{Conclusion}
We noted that heterogeneity in the beta-logistic model can better capture the temporal effects of survival and ranking at multiple horizons.
We extended the beta-logistic and its maximum likelihood estimation to linear, tree and neural models as well as characterized the convexity and properties of the learning problem. The resulting survival models give survival estimates and provably consistent rankings of who is most-at-risk at multiple time horizons. Empirical results demonstrate that the beta-logistic is an effective model in discrete time to event problems, and improves over common baselines. It seems that in practice regardless of how many attributes are considered there are still unobserved variations between individuals that influence the time to event. Further we demonstrated that we can recover posteriors effectively even when the data is very heterogeneous, and due to the speed and ease of implementation we argue that the beta-logistic is a baseline that should be considered in time to event problems in practice.
In future work, we plan to study the potential use of the beta-logistic in explore-exploit scenarios and as a viable option in reinforcement learning to model long-term consequences from near-term decisions and observations.
%
% The acknowledgments section is defined using the "acks" environment (and NOT an unnumbered section). This ensures
% the proper identification of the section in the article metadata, and the consistent spelling of the heading.
\begin{acks}
The authors would like to thank Nikos Vlassis for his helpful comments during the development of this work, and Harald Steck for his thoughtful review and guidance.% and Caitlin Smallwood for her faith that we could deliver a model that could answer many business questions.
\end{acks}
%
% The next two lines define the bibliography style to be used, and the bibliography file.
\bibliographystyle{ACM-Reference-Format}
\bibliography{sample-base}
%
% If your work has an appendix, this is the place to put it.
\begin{appendix}
\section{Beta Logistic Formulas}
\subsection{Recurrence derivation}
\label{subsection:recurrence_derivation}
This derivation is taken from Fader and Hardie \cite{fader2007project} where they use it as a cohort model (also called the shifted beta geometric model) that is not conditional on a covariate vector $x$.
We do not observe $\theta$, but its expectation given the beta prior (also called marginal likelihood) is given by:
\begin{equation*}
\begin{aligned}
\mathbb{P}(T=t|\alpha,\beta) & = \int_{0}^{1}\theta (1-\theta)^{t-1} \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}d\theta\\
& = \frac{B(\alpha+1,\beta+t-1)}{B(\alpha,\beta)}
\end{aligned}
\end{equation*}
We can write the above as:
\begin{equation*}
\mathbb{P}(T=t|\alpha,\beta) =\frac{\Gamma(\alpha+\beta)*\Gamma(\alpha+1)*\Gamma(\beta+ t - 1)}{\Gamma(\alpha)*\Gamma(\beta)*\Gamma(\alpha+\beta+t)}.
\end{equation*}
Using the property $\Gamma(z+1) = z\Gamma(z)$ leads to equations (\ref{eq:recursion_prod_p}) and (\ref{eq:recursion_prod_s}), and at $t=1$ we have
\begin{equation*}
\begin{aligned}
\mathbb{P}(T=1|\alpha,\beta) & = \frac{\Gamma(\alpha+\beta)*\Gamma(\alpha+1)*\Gamma(\beta)}{\Gamma(\alpha)*\Gamma(\beta)*\Gamma(\alpha+\beta+1)}\\
\mathbb{P}(T=1|\alpha,\beta) & = \frac{\alpha}{\alpha+\beta}
\end{aligned}
\end{equation*}
\subsection{Gradients}
Note that for machine learning libraries that do not offer symbolic computation and auto-differentiation, taking the $-log$ of equations (\ref{eq:recursion_prod_p}) and (\ref{eq:recursion_prod_s}) and differentiating leads to the following recurrence formulas for the gradient of the loss function on a given data point with respect to the output parameters $a_i$ and $b_i$ of the model considered:
\label{subsection:appendix_gradient}
\begin{equation*}
\begin{aligned}
\frac{\partial \log(\mathbb{P}(T=1))}{\partial a_i} & =\frac{\partial a}{\partial a_i} \left(\frac{\beta}{\alpha+\beta} \right)\\
\frac{\partial \log(\mathbb{P}(T=1))}{\partial b_i} & =-\frac{\partial b}{\partial b_i} \left(\frac{\beta}{\alpha+\beta}\right)
\end{aligned}
\end{equation*}
These derivatives expand as follows:
\begin{equation*}
\begin{aligned}
\frac{\partial \log(\mathbb{P}(T=t))}{\partial a_i} & = \frac{\partial \log(\mathbb{P}(T=t-1))}{\partial a_i} - \frac{\partial a}{\partial a_i}\left( \frac{\alpha}{\alpha + \beta +t-1} \right) \\
\frac{\partial \log(\mathbb{P}(T=t))}{\partial b_i} & = \frac{\partial \log(\mathbb{P}(T=t-1))}{\partial b_i} \\
& + \frac{\partial b}{\partial b_i}\left(\frac{(\alpha+1)\beta}{(\beta+t-2)(\alpha + \beta+t-1)} \right)
\end{aligned}
\end{equation*}
We can get a similar recursion for the survival function:
\begin{equation*}
\begin{aligned}
\frac{\partial \log(\mathbb{P}(T > 1))}{\partial a_i} & =-\frac{\partial a}{\partial a_i}\left(\frac{\alpha}{\alpha + \beta}\right) \\
\frac{\partial \log(\mathbb{P}(T > 1))}{\partial b_i} & =\frac{\partial b}{\partial b_i}\left(\frac{\alpha}{\alpha + \beta}\right)
\end{aligned}
\end{equation*}
%These derivatives expand as follows:
\begin{equation*}
\begin{aligned}
\frac{\partial \log(\mathbb{P}(T > t))}{\partial a_i} & =\frac{\partial \log(\mathbb{P}(T > t-1))}{\partial a_i} -\frac{\partial a}{\partial a_i}\left(\frac{\alpha}{\alpha + \beta+t-1}\right) \\
\frac{\partial \log(\mathbb{P}(T > t))}{\partial b_i} & = \frac{\partial \log(\mathbb{P}(T > t-1))}{\partial b_i} \\
& + \frac{\partial b}{\partial b_i}\left(\frac{\alpha\beta}{(\beta+t-1)(\alpha + \beta +t-1)} \right)
\end{aligned}
\end{equation*}
\subsection{Diagonal of the Hessian}
We obtain the second derivatives for the Hessian as follows:
\label{subsection:hessian}
\begin{equation*}
\begin{aligned}
\frac{\partial^2 \log(\mathbb{P}(T=1))}{\partial a_i^2} & = \frac{\partial^2 a}{\partial a_i^2}\left(\frac{\beta}{\alpha+\beta}\right)-\left(\frac{\partial a}{\partial a_i}\right)^2 \left(\frac{\alpha\beta}{(\alpha + \beta)^2} \right)\\
\frac{\partial^2 \log(\mathbb{P}(T=1))}{\partial b_i^2} & =-\frac{\partial^2 b}{\partial b_i^2}\left(\frac{\beta}{\alpha+\beta}\right)-\left(\frac{\partial b}{\partial b_i}\right)^2 \left(\frac{\alpha\beta}{(\alpha + \beta)^2} \right)
\end{aligned}
\end{equation*}
%These expand as follows:
\begin{equation*}
\begin{aligned}
\frac{\partial^2 \log(\mathbb{P}(T=t))}{\partial a_i^2} & =\frac{\partial^2 \log(\mathbb{P}(T=t-1))}{\partial a_i^2} - \left(\frac{\partial^2 a}{\partial a_i^2}\right)\left( \frac{\alpha}{\alpha + \beta +t-1} \right) \\
& - \left(\frac{\partial a}{\partial a_i}\right)^2\alpha\left( \frac{\beta+t-1}{(\alpha + \beta +t-1)^2} \right) \\
\frac{\partial^2 \log(\mathbb{P}(T=t))}{\partial b_i^2} & = \frac{\partial^2 \log(\mathbb{P}(T=t-1))}{\partial b_i^2} \\
& + \left(\frac{\partial^2 b}{\partial b_i^2}\right)\left(\frac{(\alpha+1)\beta}{(\beta+t-2)(\alpha + \beta+t-1)} \right) \\
& + \left(\frac{\partial b}{\partial b_i}\right)^2\beta\left((\alpha+1)\frac{\beta^2 - (t-2)(\alpha + t - 1)}{\left(\beta+t-2\right)^2\left(\alpha+\beta+t-1\right)^2} \right)
\end{aligned}
\end{equation*}
The survival counterparts to the above terms are also readily computed as follows:
\begin{equation*}
\begin{aligned}
\frac{\partial^2 \log(\mathbb{P}(T > 1))}{\partial a_i^2} & =- \frac{\partial^2 a}{\partial a_i^2}\left(\frac{\alpha}{\alpha+\beta}\right)-\left(\frac{\partial a}{\partial a_i}\right)^2 \left(\frac{\alpha\beta}{(\alpha + \beta)^2} \right)\\
\frac{\partial^2 \log(\mathbb{P}(T > 1))}{\partial b_i^2} & =\frac{\partial^2 b}{\partial b_i^2}\left(\frac{\alpha}{\alpha+\beta}\right)-\left(\frac{\partial b}{\partial b_i}\right)^2 \left(\frac{\alpha\beta}{(\alpha + \beta)^2} \right).
\end{aligned}
\end{equation*}
%These expand into the following:
\begin{equation*}
\begin{aligned}
\frac{\partial^2 \log(\mathbb{P}(T > t))}{\partial a_i^2} & =\frac{\partial^2 \log(\mathbb{P}(T > t-1))}{\partial a_i^2} - \left(\frac{\partial^2 a}{\partial a_i^2}\right)\left( \frac{\alpha}{\alpha + \beta +t-1} \right) \\
& - \left(\frac{\partial a}{\partial a_i}\right)^2\alpha\left( \frac{\beta+t-1}{(\alpha + \beta +t-1)^2} \right) \\
\frac{\partial^2 \log(\mathbb{P}(T > t))}{\partial b_i^2} & = \frac{\partial^2 \log(\mathbb{P}(T > t-1))}{\partial b_i^2} \\
& + \left(\frac{\partial^2 b}{\partial b_i^2}\right)\left(\frac{\alpha \beta}{(\beta+t-1)(\alpha+\beta+t-1)}\right) \\
& + \left(\frac{\partial b}{\partial b_i}\right)^2\beta\left( \alpha\frac{\beta^2 - (t-1)(\alpha + t-1) }{\left(\beta+t-1\right)^2\left(\alpha+\beta+t-1\right)^2} \right)
\end{aligned}
\end{equation*}
\section{Alternative Derivation}
Another intuitive derivation of the single-step beta-logistic is obtained by starting from the likelihood for a logistic model and modeling the probabilities with a beta distribution:
%This leads to the following likelihood:
\begin{equation*}
\begin{aligned}
L & = \prod_{i} \mathbb{P}(y_i=1|\alpha,\beta)^{y_i} (1-\mathbb{P}(y_i=1|\alpha,\beta))^{y_i-1} \\
& = \prod_{\forall y_i=1} \mathbb{P}(y_i=1| \alpha,\beta)\prod_{\forall y_i=0} (1-\mathbb{P}(y_i=1| \alpha,\beta)) \\
& = \prod_{uncensored} \mathbb{P}(T=1| \alpha,\beta)\prod_{censored} \mathbb{P}( t >=1 | \alpha,\beta).
\end{aligned}
\end{equation*}
This is exactly the survival likelihood for a 1 step beta logistic model.
\end{appendix}
\clearpage
\onecolumn
\section{Reproducibility}
We include simple python implementations of the gradient callbacks that can be passed to XgBoost or lightGBM. Note that efficient implementations of these callbacks in C++ are possible and yield orders of magnitude speedups.
\begin{lstlisting}[language=Python]
def grad_BL(alpha,beta,t,is_censored):
"""
This function computes the gradient of the beta logistic objective.
Since it is vectorized in practice for performance reasons, here we write
the non-vectorized version for readability:
"""
N = len(alpha)
g = np.zeros((N, 2))
for j in range(0,N):
if (not is_censored[j]):
#failed
g[j,0] = beta[j]/(alpha[j] + beta[j])
g[j,1] = -g[j,0]
for i in range(2,int(t[j] + 1)):
g[j,0] += -(alpha[j]/(alpha[j] + beta[j] + i - 1))
g[j,1] += beta[j]/(beta[j] + i - 2) - beta[j]/(alpha[j] + beta[j] + i - 1)
else:
#survived
g[j,:] = -alpha[j]/(beta[j] + alpha[j])
g[j,1] = -g[j,0]
for i in range(2,int(t[j] + 1)):
g[j,0] += -(alpha[j]/(alpha[j] + beta[j] + i - 1))
g[j,1] += beta[j]/(beta[j] + i - 1) - beta[j]/(alpha[j] + beta[j] + i - 1)
return g
def hess_BL(alpha,beta,t,is_censored):
"""
This function computes the diagonal of the Hessian of the beta logistic objective.
"""
N = len(alpha)
h = np.zeros((N,2))
for j in range(0,N):
h[j:] = -alpha[j]*beta[j]/((alpha[j] + beta[j])**2)
if (not is_censored[j]):
#failed
for i in range(2,int(t[j] + 1)):
h[j,0] += -alpha[j]*((beta[j] + i - 1)/(alpha[j] + beta[j] + i - 1)**2)
d = (beta[j] + i - 2)**2)*(alpha[j] + beta[j] + i - 1)**2)
h[j,1] += beta[j]*((alpha[j]+1)*(beta[j]**2-(i-2)*(alpha[j]+i-1)/d
else:
#survived
for i in range(2,int(t[j] + 1)):
h[j,0] += -alpha[j]*((beta[j] + i - 1)/(alpha[j] + beta[j] + i - 1)**2)
d = (beta[j] + i - 2)**2)*(alpha[j] + beta[j] + i - 1)**2)
h[j,1] += beta[j]*((alpha[j])*(beta[j]**2-(i-1)*(alpha[j]+i-1)/d
h.shape = (N*2)
return h
def likelihood_BL(alpha,beta,t,is_censored):
"""
This function computes beta logistic objective (likelihood : higher = better)
Since it is heavily vectorized in practice for performance reasons, we write
here the non-vectorized version for readability:
"""
p = alpha / (alpha + beta)
s = 1 - p
for j in range(0,len(alpha)):
for i in range(2,int(t[j]+1)):
p[j] = p[j] * (beta[j] + i - 2)/(alpha[j] + beta[j] + i - 1)
s[j] = s[j] - p[j]
return p * (1.0 - is_censored) + s * is_censored
\end{lstlisting}
\end{document}
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{22}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showDOI \undefined \def \showDOI #1{#1}\fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[\protect\citeauthoryear{Abadi, Barham, Chen, Chen, Davis, Dean, Devin,
Ghemawat, Irving, Isard, et~al\mbox{.}}{Abadi et~al\mbox{.}}{[n. d.]}]%
{abadi2016tensorflow}
\bibfield{author}{\bibinfo{person}{Mart{\'\i}n Abadi}, \bibinfo{person}{Paul
Barham}, \bibinfo{person}{Jianmin Chen}, \bibinfo{person}{Zhifeng Chen},
\bibinfo{person}{Andy Davis}, \bibinfo{person}{Jeffrey Dean},
\bibinfo{person}{Matthieu Devin}, \bibinfo{person}{Sanjay Ghemawat},
\bibinfo{person}{Geoffrey Irving}, \bibinfo{person}{Michael Isard},
{et~al\mbox{.}}} \bibinfo{year}{[n. d.]}\natexlab{}.
\newblock \showarticletitle{Tensorflow: a system for large-scale machine
learning.}
\newblock
\bibitem[\protect\citeauthoryear{Ben-Porath}{Ben-Porath}{1973}]%
{ben1973labor}
\bibfield{author}{\bibinfo{person}{Yoram Ben-Porath}.}
\bibinfo{year}{1973}\natexlab{}.
\newblock \showarticletitle{Labor-force participation rates and the supply of
labor}.
\newblock \bibinfo{journal}{\emph{Journal of Political economy}}
\bibinfo{volume}{81}, \bibinfo{number}{3} (\bibinfo{year}{1973}),
\bibinfo{pages}{697--704}.
\newblock
\bibitem[\protect\citeauthoryear{Berger}{Berger}{2013}]%
{berger2013statistical}
\bibfield{author}{\bibinfo{person}{James~O Berger}.}
\bibinfo{year}{2013}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{Statistical decision theory and Bayesian
analysis}}.
\newblock \bibinfo{publisher}{Springer Science \& Business Media}.
\newblock
\bibitem[\protect\citeauthoryear{Chang, Rudin, Cavaretta, Thomas, and
Chou}{Chang et~al\mbox{.}}{2012}]%
{ChangEtAl2012}
\bibfield{author}{\bibinfo{person}{Allison Chang}, \bibinfo{person}{Cynthia
Rudin}, \bibinfo{person}{Michael Cavaretta}, \bibinfo{person}{Robert Thomas},
{and} \bibinfo{person}{Gloria Chou}.} \bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{How to Reverse-Engineer Quality Rankings}.
\newblock \bibinfo{journal}{\emph{Machine Learning}} \bibinfo{volume}{88}
(\bibinfo{date}{September} \bibinfo{year}{2012}), \bibinfo{pages}{369--398}.
\newblock
Issue 3.
\bibitem[\protect\citeauthoryear{Chapelle}{Chapelle}{2014}]%
{chapelle2014modeling}
\bibfield{author}{\bibinfo{person}{Olivier Chapelle}.}
\bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Modeling delayed feedback in display advertising}.
In \bibinfo{booktitle}{\emph{Proceedings of the 20th ACM SIGKDD international
conference on Knowledge discovery and data mining}}. ACM,
\bibinfo{pages}{1097--1105}.
\newblock
\bibitem[\protect\citeauthoryear{Chen and Guestrin}{Chen and Guestrin}{2016}]%
{chen2016xgboost}
\bibfield{author}{\bibinfo{person}{Tianqi Chen} {and} \bibinfo{person}{Carlos
Guestrin}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Xgboost: A scalable tree boosting system}. In
\bibinfo{booktitle}{\emph{Proceedings of the 22nd acm sigkdd international
conference on knowledge discovery and data mining}}. ACM,
\bibinfo{pages}{785--794}.
\newblock
\bibitem[\protect\citeauthoryear{Cox}{Cox}{1972}]%
{cox1972regression}
\bibfield{author}{\bibinfo{person}{David~R Cox}.}
\bibinfo{year}{1972}\natexlab{}.
\newblock \showarticletitle{Regression models and life-tables}.
\newblock \bibinfo{journal}{\emph{Journal of the Royal Statistical Society:
Series B (Methodological)}} \bibinfo{volume}{34}, \bibinfo{number}{2}
(\bibinfo{year}{1972}), \bibinfo{pages}{187--202}.
\newblock
\bibitem[\protect\citeauthoryear{Fader and Hardie}{Fader and Hardie}{2007}]%
{fader2007project}
\bibfield{author}{\bibinfo{person}{Peter~S Fader} {and}
\bibinfo{person}{Bruce~GS Hardie}.} \bibinfo{year}{2007}\natexlab{}.
\newblock \showarticletitle{How to project customer retention}.
\newblock \bibinfo{journal}{\emph{Journal of Interactive Marketing}}
\bibinfo{volume}{21}, \bibinfo{number}{1} (\bibinfo{year}{2007}),
\bibinfo{pages}{76--90}.
\newblock
\bibitem[\protect\citeauthoryear{Fader, Hardie, Liu, Davin, and
Steenburgh}{Fader et~al\mbox{.}}{2018}]%
{fader2018project}
\bibfield{author}{\bibinfo{person}{Peter~S Fader}, \bibinfo{person}{Bruce~GS
Hardie}, \bibinfo{person}{Yuzhou Liu}, \bibinfo{person}{Joseph Davin}, {and}
\bibinfo{person}{Thomas Steenburgh}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{"How to Project Customer Retention" Revisited: The
Role of Duration Dependence}.
\newblock \bibinfo{journal}{\emph{Journal of Interactive Marketing}}
\bibinfo{volume}{43} (\bibinfo{year}{2018}), \bibinfo{pages}{1--16}.
\newblock
\bibitem[\protect\citeauthoryear{Fan}{Fan}{1991}]%
{fan1991distribution}
\bibfield{author}{\bibinfo{person}{Da-Yin Fan}.}
\bibinfo{year}{1991}\natexlab{}.
\newblock \showarticletitle{The distribution of the product of independent beta
variables}.
\newblock \bibinfo{journal}{\emph{Communications in Statistics-Theory and
Methods}} \bibinfo{volume}{20}, \bibinfo{number}{12} (\bibinfo{year}{1991}),
\bibinfo{pages}{4043--4052}.
\newblock
\bibitem[\protect\citeauthoryear{Gelman, Stern, Carlin, Dunson, Vehtari, and
Rubin}{Gelman et~al\mbox{.}}{2013}]%
{gelman2013bayesian}
\bibfield{author}{\bibinfo{person}{Andrew Gelman}, \bibinfo{person}{Hal~S
Stern}, \bibinfo{person}{John~B Carlin}, \bibinfo{person}{David~B Dunson},
\bibinfo{person}{Aki Vehtari}, {and} \bibinfo{person}{Donald~B Rubin}.}
\bibinfo{year}{2013}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{Bayesian data analysis}}.
\newblock \bibinfo{publisher}{Chapman and Hall/CRC}.
\newblock
\bibitem[\protect\citeauthoryear{Gupta, Hanssens, Hardie, Kahn, Kumar, Lin,
Ravishanker, and Sriram}{Gupta et~al\mbox{.}}{2006}]%
{gupta2006}
\bibfield{author}{\bibinfo{person}{Sunil Gupta}, \bibinfo{person}{Dominique
Hanssens}, \bibinfo{person}{Bruce Hardie}, \bibinfo{person}{William Kahn},
\bibinfo{person}{V. Kumar}, \bibinfo{person}{Nathaniel Lin},
\bibinfo{person}{Nalini Ravishanker}, {and} \bibinfo{person}{S. Sriram}.}
\bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Modeling customer lifetime value}.
\newblock \bibinfo{journal}{\emph{Journal of Service Research}}
\bibinfo{volume}{9}, \bibinfo{number}{2} (\bibinfo{year}{2006}).
\newblock
\bibitem[\protect\citeauthoryear{Heckman and Willis}{Heckman and
Willis}{1977}]%
{Heckman1977}
\bibfield{author}{\bibinfo{person}{James~J Heckman} {and}
\bibinfo{person}{Robert~J Willis}.} \bibinfo{year}{1977}\natexlab{}.
\newblock \showarticletitle{A Beta-logistic Model for the Analysis of
Sequential Labor Force Participation by Married Women}.
\newblock \bibinfo{journal}{\emph{Journal of Political Economy}}
\bibinfo{volume}{85}, \bibinfo{number}{1} (\bibinfo{year}{1977}),
\bibinfo{pages}{27--58}.
\newblock
\bibitem[\protect\citeauthoryear{Kalbfleisch and Prentice}{Kalbfleisch and
Prentice}{2011}]%
{kalbfleisch2011statistical}
\bibfield{author}{\bibinfo{person}{John~D Kalbfleisch} {and}
\bibinfo{person}{Ross~L Prentice}.} \bibinfo{year}{2011}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{The statistical analysis of failure time
data}}. Vol.~\bibinfo{volume}{360}.
\newblock \bibinfo{publisher}{John Wiley \& Sons}.
\newblock
\bibitem[\protect\citeauthoryear{Ke, Meng, Finley, Wang, Chen, Ma, Ye, and
Liu}{Ke et~al\mbox{.}}{2017}]%
{ke2017lightgbm}
\bibfield{author}{\bibinfo{person}{Guolin Ke}, \bibinfo{person}{Qi Meng},
\bibinfo{person}{Thomas Finley}, \bibinfo{person}{Taifeng Wang},
\bibinfo{person}{Wei Chen}, \bibinfo{person}{Weidong Ma},
\bibinfo{person}{Qiwei Ye}, {and} \bibinfo{person}{Tie-Yan Liu}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Lightgbm: A highly efficient gradient boosting
decision tree}. In \bibinfo{booktitle}{\emph{Advances in Neural Information
Processing Systems}}. \bibinfo{pages}{3146--3154}.
\newblock
\bibitem[\protect\citeauthoryear{Li, Chu, Langford, Moon, and Wang}{Li
et~al\mbox{.}}{2012}]%
{li2012unbiased}
\bibfield{author}{\bibinfo{person}{Lihong Li}, \bibinfo{person}{Wei Chu},
\bibinfo{person}{John Langford}, \bibinfo{person}{Taesup Moon}, {and}
\bibinfo{person}{Xuanhui Wang}.} \bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{An unbiased offline evaluation of contextual bandit
algorithms with generalized linear models}. In
\bibinfo{booktitle}{\emph{Proceedings of the Workshop on On-line Trading of
Exploration and Exploitation 2}}. \bibinfo{pages}{19--36}.
\newblock
\bibitem[\protect\citeauthoryear{MacKay}{MacKay}{2003}]%
{mackay2003information}
\bibfield{author}{\bibinfo{person}{David~JC MacKay}.}
\bibinfo{year}{2003}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{Information theory, inference and learning
algorithms}}.
\newblock \bibinfo{publisher}{Cambridge university press}.
\newblock
\bibitem[\protect\citeauthoryear{Miller}{Miller}{2015}]%
{miller_2015}
\bibfield{author}{\bibinfo{person}{Evan Miller}.}
\bibinfo{year}{2015}\natexlab{}.
\newblock \bibinfo{title}{Bayesian AB Testing}.
\newblock
\newblock
\urldef\tempurl%
\url{http://www.evanmiller.org/bayesian-ab-testing.html#cite1}
\showURL{%
\tempurl}
\bibitem[\protect\citeauthoryear{Rostykus and Raimond}{Rostykus and
Raimond}{2018}]%
{rostykusvectorflow}
\bibfield{author}{\bibinfo{person}{Beno{\^\i}t Rostykus} {and}
\bibinfo{person}{Yves Raimond}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{vectorflow: a minimalist neural-network library}.
\newblock \bibinfo{journal}{\emph{SysML}} (\bibinfo{year}{2018}).
\newblock
\bibitem[\protect\citeauthoryear{Rudin, Waltz, Anderson, Boulanger,
Salleb-Aouissi, Chow, Dutta, Gross, Huang, Ierome, Isaac, Kressner,
Passonneau, Radeva, and Wu}{Rudin et~al\mbox{.}}{2012}]%
{RudinEtAl12}
\bibfield{author}{\bibinfo{person}{Cynthia Rudin}, \bibinfo{person}{David
Waltz}, \bibinfo{person}{Roger~N. Anderson}, \bibinfo{person}{Albert
Boulanger}, \bibinfo{person}{Ansaf Salleb-Aouissi}, \bibinfo{person}{Maggie
Chow}, \bibinfo{person}{Haimonti Dutta}, \bibinfo{person}{Philip Gross},
\bibinfo{person}{Bert Huang}, \bibinfo{person}{Steve Ierome},
\bibinfo{person}{Delfina Isaac}, \bibinfo{person}{Arthur Kressner},
\bibinfo{person}{Rebecca~J. Passonneau}, \bibinfo{person}{Axinia Radeva},
{and} \bibinfo{person}{Leon Wu}.} \bibinfo{year}{2012}\natexlab{}.
\newblock \showarticletitle{Machine Learning for the {N}ew {Y}ork {C}ity Power
Grid}.
\newblock \bibinfo{journal}{\emph{{IEEE} Transactions on Pattern Analysis and
Machine Intelligence}} \bibinfo{volume}{34}, \bibinfo{number}{2}
(\bibinfo{date}{February} \bibinfo{year}{2012}), \bibinfo{pages}{328--345}.
\newblock
\bibitem[\protect\citeauthoryear{Savage}{Savage}{1994}]%
{Savage1994}
\bibfield{author}{\bibinfo{person}{Richard~P. Savage}.}
\bibinfo{year}{1994}\natexlab{}.
\newblock \showarticletitle{The Paradox of Nontransitive Dice}.
\newblock \bibinfo{journal}{\emph{The American Mathematical Monthly}}
\bibinfo{volume}{101}, \bibinfo{number}{5} (\bibinfo{year}{1994}).
\newblock
\bibitem[\protect\citeauthoryear{W.~Vaupel and Yashin}{W.~Vaupel and
Yashin}{1985}]%
{vaupel1985}
\bibfield{author}{\bibinfo{person}{James W.~Vaupel} {and}
\bibinfo{person}{Anatoliy Yashin}.} \bibinfo{year}{1985}\natexlab{}.
\newblock \showarticletitle{Heterogeneity's Ruses: Some Surprising Effects of
Selection on Population Dynamics}.
\newblock \bibinfo{journal}{\emph{The American statistician}}
\bibinfo{volume}{39} (\bibinfo{date}{09} \bibinfo{year}{1985}),
\bibinfo{pages}{176--85}.
\newblock
\urldef\tempurl%
\url{https://doi.org/10.1080/00031305.1985.10479424}
\showDOI{\tempurl}
\end{thebibliography}