Additive Logistic Models as Interpretable Likelihood-Ratio Scores for AUC-Based Classification
Extracted source text ยท arXiv 1511.04803
%%
%% Copyright 2007, 2008, 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version. The latest version of this license is in
%% http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with harvard style bibliographic references
%% SP 2008/03/01
\documentclass[authoryear]{elsarticle}
%% Use the option review to obtain double line spacing
%% \documentclass[authoryear,preprint,review,12pt]{elsarticle}
%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times,authoryear]{elsarticle}
%% \documentclass[final,1p,times,twocolumn,authoryear]{elsarticle}
%% \documentclass[final,3p,times,authoryear]{elsarticle}
%% \documentclass[final,3p,times,twocolumn,authoryear]{elsarticle}
%% \documentclass[final,5p,times,authoryear]{elsarticle}
%% \documentclass[final,5p,times,twocolumn,authoryear]{elsarticle}
%% For including figures, graphicx.sty has been loaded in
%% elsarticle.cls. If you prefer to use the old commands
%\usepackage{epsfig, graphics}
%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
\usepackage{amsthm}
\usepackage{amsmath}
%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers.
%% \usepackage{lineno}
%\journal{}
\begin{document}
\begin{frontmatter}
%% Title, authors and addresses
%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for theassociated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for theassociated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for theassociated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}
\title{Performance and Interpretation of Classification Models}
%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{}
%% \address[label1]{}
%% \address[label2]{}
\author{Yuan-chin Ivan Chang and Chih-Li Sung}
\address{Institute of Statistical Science,
Academia Sinica,
128, Academia Road Section 2, Taipei
Taiwan 11529}
\begin{abstract}
%% Text of abstract
Classification is a common statistical task in many areas. In order to ameliorate the performance of the existing methods, there are always some new classification procedures proposed. These procedures, especially those raised in the machine learning and data-mining literature, are usually complicated, and therefore extra effort is required to understand them and the impacts of individual variables in these procedures.
%
However, in some applications, for example, pharmaceutical and medical related research, future developments and/or research plans will rely on the interpretation of the classification rule, such as the role of individual variables in a diagnostic rule/model. Hence, in these kinds of research, despite the optimal performance of the complicated models, the model with the balanced ease of interpretability and satisfactory performance is preferred.
%
The complication of a classification rule might diminish its advantage in performance and become an obstacle to be used in those applications.
%
In this paper, we study how to improve the classification performance, in terms of area under the receiver operating characteristic curve of a conventional logistic model, while retaining its ease of interpretation. The proposed method increases the sensitivity at the whole range of specificity and hence is especially useful when the performance in the high-specificity range of a receiver operating characteristic curve is of interest. Theoretical justification is presented, and numerical results using both simulated data and two real data sets are reported.
\end{abstract}
\begin{keyword}
%% keywords here, in the form: keyword \sep keyword
%% PACS codes here, in the form: \PACS code \sep code
%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)
Area under curve; generalized additive model; logistic regression model; ROC curve
\end{keyword}
\end{frontmatter}
%% \linenumbers
%% main text
\section{Introduction}
Due to the predictive nature of classification models, the predictive power is usually the most important factor for assessing a classifier/diagnosis procedure. This is especially the case in the scenarios of disease screening or survey research, where an efficient classification rule that can effectively distinguish the diseased from normal subjects is preferred. %the performance is the major factor for assessing classification rules.
%
However, we are also aware that the performance should not be the solo measure for assessing a classifier when the model interpretation and the impacts of factors are of interest. For example, in some medical/pharmaceutical research, we want to learn the impacts of biomarkers in a classification rule. Because this information might also imply their associations with the studied disease, it could be useful for drug and therapy developments. Hence, in this kind of research, besides the diagnostic power, the ability of to identify useful biomarkers/variables or the ease of model interpretation is also an important criterion for evaluating a classification rule.
In modern classification literature, we can easily find many powerful classification procedures, such as the support vector machine \citep[see][]{SVM-intro}, which can provide high classification accuracy. Yet, we also know that when these types of powerful and complicated classification rules are used, extra effort is required to interpret their rules/models, and to dig out the impact information of individual variables/biomarkers from them is even more difficult.
On the other hand, simple conventional statistical models, such as linear logistic models, can usually be easily interpreted; nevertheless, they might not have as good classification performance as those modern methods, especially when the association between diseases and variables is complicated.
A nonlinear model can usually fit the observed data better than its linear model counterpart. However, can it also assure us good prediction performance in general classification problems? As far as we know, there is still a lack of theoretical justification in the literature to guarantee this.
In fact, the inconsistency between the fitted model and the prediction-based model building has been reported in \citet{Tian07}.
Of course, if the model is true, then fitting observations may better insure this model against bad predictions.
However, in practical situations, the true model is usually unknown.
Then, to overfit the observed data or training samples with a complicated model cannot guarantee good classification/prediction performance.
%There are some discussions about the fitting-based and prediction-based model selections in the literature.\cite{Tian07}
From practical perspectives, to balance the classification performance and the ease of model interpretation is important,
especially in the studies that focus on providing detailed model information for further research and developments.
This motivates us to find a way to boost the performance of the conventional model and retain its ease of model interpretation.
The assessment measures play an important role in classification/diagnostic research \citep[see][]{pepe2004statistical}, and using a different assessment measure commonly results in selecting a different classifier.
Because the accuracy of a binary classifier can easily be affected by the selected cutting point, and because using training samples to decide a cutting point might easily cause overfitting, a cross-validation procedure is usually used to prevent this situation. However, the cross-validation method is computationally intensive.
Thus, threshold independent measures, such as the receiver operating characteristic (ROC) curve, are usually preferred to accuracy \citep{pepe2004statistical}.
The area under ROC curve (AUC) is a popularly used summarization statistic of the ROC curve, and in this study, we will use it as the performance measure.
%
For a binary classification problem, it has been proved in \citet{eguchi2002class} that a logistic-type classification function will achieve the maximum AUC under certain conditions.
Moreover, as stated in \citet{hastie, GAM}, the general additive model (GAM) is a method of approximating the true model and is a flexible statistical method that can be used to characterize and identify nonlinear regression effects.
Many advantages of GAM have been intensively discussed.
%\citep[see][]{hastie-tibshirani-friedman2001, wood2004stable, Wood08}.
Among them, a commonly mentioned one is its ease of model interpretability \citep{hastie, GAM, Wood06}.
%
\iffalse
The general additive models (GAM) \citep{hastie} are flexible statistical methods that may be used to characterize and identify nonlinear regression effects.
Many advantages of GAM have been intensively discussed \citep{hastie-tibshirani-friedman2001, wood2004stable, Wood08}, and a commonly mentioned one is its ease of model interpretability \citep{hastie, GAM, Wood06}.
%a general comment is that ``When to interpret a nonparametric model is very difficult, and to fit a linear model is not possible, the GAM is a good alternative because it retains some of the interpretability of the linear model by assuming additive effect.''
Moreover, the GAM is a method of approximating the true model \citep{hastie, GAM}, and it has been proved that a logistic type classification will achieve the maximum AUC \citep{eguchi2002class} under certain condition.
The results mentioned above motivate us to study how to use a general additive logistic function as a classification model such that model interpretation ability is retained, and a better classification performance is achieved.
\fi
%
Hence, in this paper, a general additive logistic type of classification function is used to boost the classification performance of conventional models in terms of AUC. Theoretical justification is presented, and some suggestions for fitting a general additive logistic model are discussed.
The rest of this paper is organized as follows.
We start with a brief review of the ROC curve, AUC, and GAM. Then, we will show the relations between the ROC curve and the logistic regression models, and how the AUC of a logistic model can be improved with a general additive logistic model.
%
It is followed by the numerical results using both synthesized data and two real data sets.
Fitting a GAM model relies on some iterative algorithms, and the different approaches used in the algorithms may affect the fitting results.
Hence, the computational methods of GAM are compared, and some suggestions based on the empirical results are given in the Summary. The information on algorithms and the corresponding packages used here are described in the Appendix.
%\vspace{-0.18in}
\section{Review and Methodology}
\subsection{ROC cuver and AUC}
The ROC curve and AUC are important statistical tools for evaluating binary classifiers \citep[see][]{pepe2004statistical}.
Let $Z$ be the score of a binary classification function; then they can be briefly described as follows.
Assume that the greater the value of $Z$, the higher the probability of a subject being diagnosed as a diseased one.
Suppose $c \in R$ is a threshold and the subject with $Z>c$ will be classified as diseased.
Then, the sensitivity and specificity for such a cutting point $c$ are defined as $Se(c)=pr(Z > c)$ and $Sp(c)=pr(Z>c)$, respectively. It follows that the corresponding ROC curve is defined as a plot of {$\{(1-Sp(c), Se(c)): -\infty<c<\infty\}$}, and its AUC is then equal to {$\int_0^1 Se(Sp^{-1}(t))dt$.} That is, the plot of an ROC curve takes all possible cutting points into consideration, and the AUC of it can be viewed as a summarization of ROC curve.
The detailed properties of the ROC curve and AUC can be easily found in the literature and textbooks, such as \citet{pepe2004statistical}.
\subsection{General additive logistic models}
Let $x_1$, $x_2$, $\ldots, x_p$ be predictors and $y \in \{0, 1\}$ be a binary response variable. A GAM has a form as
\begin{equation}\label{gam_formula}
E(y|x_1, x_2,\ldots, x_p)=\alpha+f_1(x_1)+f_2(x_2)+\ldots+f_p(x_p),
\end{equation}
where $f_j$'s are some smooth functions.
Thus, it can be viewed as an additive extension of the family of generalized linear models \citep{GAM}.
Each smooth function in \eqref{gam_formula} can be fitted using a scatterplot smoother, such as a cubic smoothing spline or kernel smoother.
As a special case of the generalized linear models, a general additive logistic model, under the GAM framework, is formulated as
\begin{equation}\label{gam_logistic_formula}
\log\left(\frac{p(x)}{1-p(x)}\right)=\alpha+f_1(x_1)+f_2(x_2)+\ldots+f_p(x_p),
\end{equation}
where $p(x)=P(y=1|x)$ and $f_j$'s are some unknown smooth functions.
The smooth functions of GAMs are estimated based on the observations with some iterative procedures.
Due to the additive assumption, these smooth functions can always provide the relations between the variables and the disease.
The non-parametric form of $f_j$ gives us some flexibility for model fitting, and the additive form of GAM retains much of the ability of model interpretation \citep{hastie-tibshirani-friedman2001}.
Besides these properties, there are more classification-related advantages of the additive logistic models discussed in \citet{Friedman2k}.
In many medical diagnostic situations, to learn the relationship between the sensitivity and specificity of classifiers is important.
An ROC curve and its AUC can provide more information before a threshold is determined. Thus, despite the accuracy of a classification rule, they are commonly used measures in these kinds of research.
Below, we show how the classification power of a conventional logistic model can easily be improved
by an additive logistic model counterpart with only a little extra computational effort.
%\vspace{-0.6cm}
\subsection{AUC of additive logistic models}\label{proof_section}
Let $y$ be a binary response variable as before.
Suppose that $y = 0$ {or} $1$ are labels of two classes in a binary classification problem. Let $x\in R^p$ be the vector of measurements of a subject and $g_0(x)$ and $g_1(x)$ be the conditional probability density functions of $x\in R^p$, given $y=0$ and $y=1$, respectively.
Let $\pi_0$ and $\pi_1$, with $\pi_0+\pi_1=1$, denote the probabilities of $y=0$ and $y=1$, accordingly. That is, $\pi_0$ and $\pi_1$ are the proportions of two classes in the whole population.
%
It follows that for a given $x$, the log-odd ratio of two classes, say $\lambda(x)$, can be written as a function of $x$:
\begin{align}\label{log-like-ratio}
\lambda(x)=\log\frac{p(x)}{1-p(x)}=\log\frac{\pi_1}{\pi_0}+\log\frac{g_1(x)}{g_0(x)}.
\end{align}
As stated before, an ROC curve is a plot true positive rate (sensitivity) against the false positive rate (1-specificity).
If we treat the false positive and false negative as Type I and Type II errors, respectively,
then an ROC curve can also be viewed as a plot of Type II errors versus Type I errors at different critical values.
%
Moreover, from the Neyman-Pearson lemma, we have that for a given Type I error, the likelihood ratio test is the most powerful test for testing the null hypothesis $y=0$ versus the alternative hypothesis $y=1$.
Hence, under this hypotheses testing framework, and using the Neyman-Pearson lemma,
it is proved that $\lambda(x)$ defined in \eqref{log-like-ratio} is a statistic that maximizes the height of the ROC curve at all levels of specificity. %\cite{eguchi2002class}.
That is, if $\lambda(x)$ is used as a classification function, then its corresponding ROC curve will dominate the ROC curves of all other binary classification functions and therefore has the maximum AUC. %\citep{eguchi2002class}.
Based on this fact, \citet{eguchi2002class} showed that the maximum AUC can only be achieved when the true model for the density ratio, ${g_1(x)}/{g_0(x)}$, is known.
However, both $g_0(x)$ and $g_1(x)$ are usually unknown in practice, and so is $\lambda(x)$.
Thus, among all possible classification functions based on the observations, the one that can best approximate $\lambda(x)$ will have the largest maximum AUC. It follows that when a conventional logistic model is a true model, then the classification function based on it will have the maximum AUC. Therefore, in this case there is no room for further improvement.
Otherwise, if the prediction power is the only concern, then one can always find a more complicated model for approximating the likelihood ratio in order to have better classification performance.
On the other hand, if the final model is too difficult to be interpreted, then it provides little information about the relation between responses and the variables of interest.
Thus, these types of models will not be useful for identifying the impacts of individual variables on the response.
However, in many situations, to identify the key variables is also an important task.
For example, in pharmaceutical and medical research, this information is highly valued for developing new drugs or therapies.
It is well-known that a general additive model will usually fit data well. In addition, authors suggest that to use an additive logistic model to fit data is a good strategy in practice, as it will retain some ease of model interpretation \citep{Friedman2k}.
However, it is also pointed out in the literature that a better-fitting model cannot guarantee a better prediction \citep{Tian07} in a general sense, and as far as we know, there is still no theoretical proof that a better-fitting model will also have a larger AUC. In below, we show that a general additive model will have a larger AUC than its linear logistic model counterpart will.
\paragraph{Proof}
Let $\lambda(x)$ be the true log-odds ratio as before.
If a linear logistic model is used to approximate the log odds-ratio, then we assume that $\log[{p(x)}/{1-p(x)}]=\beta'x$, where $\beta$ is the unknown regression parameters to be estimated.
Similarly, if an additive logistic model is used, then we set $\log[{p(x)}/{1-p(x)}]=\Sigma_i f_i$, where $f_i$'s are some smooth functions as in \eqref{gam_logistic_formula}.
Let $u$, $u'$ and $u''$ be three constants and define
%\[
$A=\{x:\lambda(x)>u''\}$, $B=\{x:\Sigma_if_i>u'\}$, $C=\{x:\beta'x>u\}$
%\]
as the corresponding critical regions of the likelihood ratio tests based on $\lambda(x)$, $\Sigma_if_i$ and $\beta'x$, respectively, such that
\begin{equation}\label{type-I}
\int_{A}{g_0(x)dx}=\int_{B}{g_0(x)dx}=\int_{C}{g_0(x)dx}.
\end{equation}
That is, all three test statistics, $\lambda(x)$, $\Sigma_if_i$ and $\beta'x$, have the same Type I error.
Then, by the Neyman-Pearson lemma, we have
%\begin{equation}\label{gam_inequ_new1}
% \int_{A}{g_1(x)dx}-\int_{B}{g_1(x)dx}\geq \epsilon_1,
%\end{equation}
%and
\begin{equation}\label{linear_inequ_new1}
\int_{A}{g_1(x)dx}-\int_{C}{g_1(x)dx} \geq 0,
\end{equation}
and the equality in \eqref{linear_inequ_new1} holds only when the linear logistic model is the true model for the log odds-ratio.
If this linear logistic model is not the true model, then there is an $\epsilon_1 > 0$ such that
\begin{equation}\label{linear_inequ_new2}
\int_{A}{g_1(x)dx}-\int_{C}{g_1(x)dx} \geq \epsilon_1,
\end{equation}
%
\iffalse
If inequality
\begin{align}\label{ineq}
\int_{B}{g_1(x)dx} \geq \int_{C}{g_1(x)dx}
\end{align}
holds, then it follows from \eqref{gam_inequ_new1} and \eqref{linear_inequ_new1}, we have proved that the AUC of an additive logistic model defined by $\Sigma_if_i$ is greater than or equal to that of a linear logistic model defined by $\beta'x$.
It is clear that to prove \eqref{ineq} is equivalent to show that $C \subseteq B$.
However, in general there is no guarantee that $C \subseteq B$, when $f_i$'s are just some nonlinear or additive extensions of the original linear logistic model.
In below, we show that if $\Sigma_if_i$ satisfy certain conditions, then the inequality in \eqref{ineq} holds, and thus the proof is complete.
\fi
%
Suppose that $x_j$'s are bounded; then the estimates of $f_i$'s, say $f_i^*$'s, are chosen to minimize $E[(\lambda(x)-\Sigma_if^\ast_i)^2]$
subject to the constraint $E(f^\ast_i)=0$. Then, under a common smoothness assumption on $f^\ast_i$, Stone \citep{stone1985additive} showed that %under $L^2$-norm,
the rate of convergence for spline estimates of $\Sigma_if^\ast_i$ is $n^{-(2p)/(2p+1)}$,
where $n$ is the sample size and $p$ is a measure of the smoothness of $f$.
%It implies that a splines estimate of $\Sigma_if^\ast_i$ will approximate to $\lambda(x)$ sufficiently well when sample size is large enough.
Thus, $E[(\Sigma_if_i(X)- \lambda(X))^2]$ will converge to 0 as the sample size becomes large. %which also implies that the differences of set $A$ and set $B$ becomes a empty set.
Because \eqref{type-I} holds, it implies that for any given $\epsilon_2 \geq 0$, there is a large enough $n$ such that
\begin{equation}\label{gam_inequ_new2}
\epsilon_2 \geq\int_{A}{g_1(x)dx}-\int_{B}{g_1(x)dx} \geq 0.
\end{equation}
Because $\epsilon_2$ can be arbitrarily small as $n$ goes to infinity, equations \eqref{linear_inequ_new1} and \eqref{gam_inequ_new2} together imply that as the sample size $n$ is large enough,
\begin{equation}\label{difference}
\int_{B}{g_1(x)dx}-\int_{C}{g_1(x)dx}\geq \epsilon_1-\epsilon_2 > 0.
\end{equation}
%It follows from Stone\cite{stone1985additive}, when the sample size is sufficiently large, $\epsilon_3$ goes to zero.
%This implies that $\epsilon_2-\epsilon_3 >0$ for large enough $n$.
It implies that for a large enough $n$, we have
\begin{displaymath}
\int_{B}{g_1(x)dx} - \int_{C}{g_1(x)dx} > 0.
\end{displaymath}
That is, when the sample size is large enough,
there is a spline estimate, $\Sigma_if_i$, of $\lambda(x)$ such that
\begin{equation}\label{summary}
\int_{A}{g_1(x)dx}\geq\int_{B}{g_1(x)dx} > \int_{C}{g_1(x)dx}.
\end{equation}
Let $\delta(u)$ be a function of $u$, $-\infty<\delta(u)<\infty$ such that $\text{pr}\{\Sigma_if_i>u|y=0\}=\text{pr}\{\beta'x>u+\delta(u)|y=0\}$. Therefore, $\text{pr}\{\Sigma_if_i>u|y=1\}\geq\text{pr}\{\beta'x>u+\delta(u)|y=1\}$
as $\text{pr}\{\Sigma_if_i>u|y=0\}=\text{pr}\{\beta'x>u+\delta(u)|y=0\}$.
Let AUC$_{\text{GAM}}$ denote the AUC of the general additive logistic model based on $\Sigma_if_i$, and let AUC$_{\text{GLM}}$ denote the AUC of the linear logistic regression model based on $\beta'x$.
Then, the rest of the proof follows via similar arguments of that in \citet{eguchi2002class}. That is, it is shown that
\begin{flalign*}
\text{AUC}_{\text{GAM}}=& \int_{u=+\infty}^{-\infty}{\text{pr}\{\Sigma_if_i>u|y=1\}d(\text{pr}\{\Sigma_if_i>u|y=0\})}\\
& =\int_{u=+\infty}^{-\infty}{\text{pr}\{\Sigma_if_i>u|y=1\}d(\text{pr}\{\beta'x>u+\delta(u)|y=0\})}\\
&\geq\int_{u=+\infty}^{-\infty}{\text{pr}\{\beta'x>u+\delta(u)|y=1\}d(\text{pr}\{\beta'x>u+\delta(u)|y=0\})}=\text{AUC}_{\text{GLM}}.
\end{flalign*}
This completes that proof that as $n$ is large, the general additive logistic model based on $\Sigma_if_i$ has a larger AUC than the linear logistic model with $\beta'x$.
%
The above result says that unless the linear logistic model above is the true model, which leaves no room for improvement, there is always a general additive logistic model that has a larger AUC than that of the original linear logistic model when the sample size is large.
Moreover, because \eqref{summary} holds for all classification thresholds, it implies that the general additive logistic model has larger sensitivity than its corresponding linear logistic model does at any specificity level.
That is, the general additive logistic model actually also has a larger partial AUC than the linear logistic model does at any given range of specificity. This is a very useful feature, especially in rare disease diagnosis or other similar situations where the high specificity of a classification model is a must.
\section{Empirical Study}
We report the numerical results using both synthesized and real data sets.
There are several numerical algorithms available in the R platform that can be used to fit a general additive logistic model with different options. According to the computational approaches adopted in these algorithms, there are three categories:
\begin{itemize}
\item[](a) back-fitting method (see also \citet{GAM}, Chapter 4),
\item[](b) simultaneous estimation method (estimating all components with optimization in smoothing parameter space \citep{marx1998direct, wood2004stable}, and
\item[](c) likelihood-based boosting method \citep{tutz2006generalized}.
\end{itemize}
The computational efficiency depends on many factors such as the dimensionality of explanatory variables of the data set, sample size and so on.
Basically, we found that none of these three types of methods can dominate the other two in all situations.
Hence, in order to provide useful information for practitioners to choose a suitable one for their needs, the advantages and disadvantages of using these methods in different situations are discussed.
The results of fitting a linear logistic regression with a conventional method are used as baseline models,
and the results of the general additive logistic models using different fitting algorithms will be compared with these results.
(The variable selection is not the focus in this paper; the variable selection schemes will be used only when there is such an option available in the corresponding packages.)
The following abbreviations denote the algorithms and the options used in our numerical studies.
%
\begin{enumerate}
\item \emph{GLM}: Fitting a {\it linear logistic regression model} with all predictors
\item \emph{GLM\_step}: Fitting a {\it linear logistic regression model} with a backward elimination method for variable selection
\item \emph{Backfitting}: Fitting a {\it general additive logistic model} using a back-fitting algorithm
\item \emph{Backfitting\_step}: Fitting a {\it general additive logistic model} using a back-fitting algorithm with the AIC option for stepwise selection of model components and degrees of freedom
\item \emph{mgcv}: Simultaneously estimating all components with smoothing parameter optimization when fitting a {\it general additive logistic model}
\item \emph{mgcv\_step}: Similar to \emph{mgcv} with automatic selection of smoothing parameter using the AIC criterion
\item \emph{GAMBoost}: A boosting procedure with the number of boosting steps selected with the AIC criterion
\end{enumerate}
Because fitting a GAM relies on the iterated algorithms and some parameter-setting in the algorithms need to be set in advance, there are many suggestions for the parameter setting in the algorithms in \citet{Wood08, hastie-tibshirani-friedman2001, tutz2006generalized}.
Here, we follow the suggestions of \citet{tutz2006generalized} by apply a scaling factor 1.4 to the effective degrees of freedom of \emph{mgcv},
and we multiply the degrees of freedom of both \emph{GLM} and \emph{backfitting} methods by 1.4.
For the \emph{GAMBoost} has a variable selection option during its boosting procedure, so we use the default variable selection scheme in \emph{GAMBoost}.
All numerical studies are conducted using the R language on a 64-bit Linux PC cluster with Xeon Octa-Core and Hyper-Threading E5-2670 2.6 GHz CPU with 256GB of RAM.
The technical details of the packages used here, please refer to their original manuals in R.
In addition to AUCs, the computational times of using these methods in different model-fitting scenarios in our numerical studies are reported. The fitting methods are described in the Appendix.
We just use the variable selection option when it is available in a package; however, we will not discuss much about it.
For the details of the variable selection option, please refer to their original papers, which can be found in the documents of the corresponding packages.
Instead, we emphasize the improvement of AUCs when a general additive logistic type classification function is used and its interpretation.
\subsection{Synthesized data}\label{simulation}
We consider two different dimensions ($d=5$ and $d=10$) of models.
To simulate different kinds of model complexities, we use two sets of effective variables as described in equations \eqref{easy_func} and \eqref{hard_func} below:
\begin{align}
\hbox{Set 1:~} & g_1 = x,\quad g_2 = 2x^2,\quad g_3 = \sin(5x), \label{easy_func}\\
\hbox{Set 2:~} & g_1 = 2(-x^3+1),\quad g_2 = 3\exp(-5x^2),\quad g_3 = 4\log(1+x^2). \label{hard_func}
\end{align}
The others variables, $x_4, \ldots, x_d$ of the model are generated from a uniform distribution $U[-1,1]$.
That is, in all cases, only Variable Set 1 and Set 2 above have contributions to their corresponding responses, and
the other variables $x_4, \ldots, x_d$ will have no contributions at all.
In our studies, the true log-odds function of models with Set 1 and Set 2 variables have the same form below:
\begin{displaymath}
\eta = 3(-0.7+g_1(x_1) + g_2(x_3) + g_3(x_5)).
\end{displaymath}
The true probability vector $p=(p_1,...,p_n)'$ is calculated with $p_i=\exp(\eta_i)/(1+\exp(\eta_i))$ for each $i$, and
the binary response vector $y=(y_1,...,y_n)'$ is generated using a Bernoulli distribution with success probability $p_i$ for each $i=1, \ldots, n$.
There are 100 repetitions for each case. For each run, we separately generated 100 training samples and 1000 testing samples.
%
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{simulated_data_AUC}
\caption{Box-plots of AUCs with 100 repetitions in four difference scenarios: (a) $d=5$ with Equation \eqref{easy_func}, (b) $d=5$ with Equation \eqref{hard_func}, (c) $d=10$ with Equation \eqref{easy_func} and (d) $d=10$ with Equation \eqref{hard_func}.}
\label{figure1}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{ROC_simulated_data}
\caption{The average ROC curve at the left-upper corner of the $[0,1]\times[0,1]$ unit square for each method. The curve of each method is based on 100 repetitions, and data are generated from 4 different situations: (a) $d=5$ with Equation \eqref{easy_func}, (b) $d=5$ with Equation \eqref{hard_func}, (c) $d=10$ with Equation \eqref{easy_func} and (d) $d=10$ with Equation \eqref{hard_func}. The vertical bars in these plots indicate the 95\% point-wise confidence intervals for each curve.}
\label{figure10}
\end{figure}
%
Figure \ref{figure1} shows the box-plots of AUCs based on 100 replications for different models using different fitting methods described before.
For the model with Set 1 as its effective variables, all fitting methods for the general additive model have larger AUCs than the conventional linear logistic models do (with and without variable selection option). For the model with the effective variable Set 2, the advantage of the general additive logistic model is not as significant as the previous case, which is due to the complexity of the true model such that both the linear and general additive models cannot approximate the true model well. However, the additive logistic model using \emph{backfitting} and \emph{GAMBoost} still outperforms the linear logistic model.
{%\color{red}
Figure \ref{figure10} are plots of the average ROC curves in the left-upper corner of the unit-square, $[0,1]\times[0,1]$. We can clearly see that there are two groups of ROC curves in each picture, and the ROC curves of the general additive model, with different fitting methods, are always higher than that of logistic models (GLM and GLM\_step). The general additive model fitted with different algorithms is essentially the same model unless their default variable selection option is used, which may introduce some variation due to their variable selection schemes.
(Note that GLM denotes the result of fitting a conventional linear logistic model, and GLM\_step denotes the result of a similar fitting procedure but with a stepwise variable selection option.)
}
Figure \ref{figure3} shows the box-plots of computational times of different methods. When $d=5$, the \emph{backfitting} method uses the shortest time among all methods in two models. When $d=10$, the \emph{GAMBoost} method becomes very competitive, especially when the model is complicated as in \eqref{hard_func} (see Figure \ref{figure3}(d)).
Among different fitting methods for the general additive logistic model, the \emph{backfitting} and \emph{GAMBoost} are more computationally efficient than are the \emph{mgcv} methods with and without step-wise variable selection.
Based on the simulation results, the \emph{GAMBoost} \emph{backfitting} methods are ``cost-effective'' in terms of their gains in AUCs and the little cost of extra computational time used compared to that of fitting a conventional linear logistic model with \emph{GLM}-based algorithms.
There are more discussions and comparisons of the computational-related issues for algorithms for fitting a logistic model and GAM that can be found in \citet{Minka07, binder-tutz08}.
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{simulated_data_Time_cost}
\caption{The CPU times for fitting models using different methods, based on 100 repetitions, with data generated from (a) $d=5$ with Equation \eqref{easy_func}, (b) $d=5$ with Equation \eqref{hard_func}, (c) $d=10$ with Equation \eqref{easy_func} and (d) $d=10$ with Equation \eqref{hard_func}.}
\label{figure3}
\end{figure}
\subsection{Real data examples}
We also apply the proposed method and algorithms discussed above to two real data sets, Parkinson's disease \citep{little2007exploiting} and cardiotocography \citep{Bache+Lichman:2013}, for illustration purposes.
We compared the performances of methods using a general additive logistic model to the results obtained via fitting a linear logistic model.
The main purpose of using these two data sets is to assess the improvement in AUCs when replacing a linear logistic with a general additive logistic model. Hence, for the details of these two data sets, please refer to their original papers \citep{little2007exploiting, Bache+Lichman:2013}. Brief information about these two data sets is summarized below.
\paragraph{Parkinson's disease data set} The original purpose of this data set is to find a classification rule to discriminate healthy people from the Parkinson's disease (PD) patients based on some biomedical voice measurements.
There are 31 persons in this study, and among them, 23 are patients.
The study includes 195 voice recordings from these 31 individuals with different numbers of replications of each individual.
There are 23 variables for each recoding.
The binary code indicates the voice recording is from a diseased or a normal subject.
The other 22 predictive variables consist of the records of several voice measurements, such as average vocal fundamental frequency, variation in amplitude and so on.
For illustration purposes, we treat 195 voice recordings as 195 individual samples.
Thus, among these 195 voices, there are 147 voice recordings from PD subjects and 48 from healthy subjects.
That is, we treat it as a data set with 195 samples (147 positive and 48 negative cases) with 22 variables, and the target becomes to build a classification rule to distinguish these two types of recordings.
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{Parkinsons_figure}
\caption{(a) AUC and (b) computational cost time measured by each fitting method with 100 resampling data from parkinson's data set.}
\label{figure5}
\end{figure}
We follow the common training-test framework in machine learning and classification literature.
Each time, 90\% of samples are randomly selected from these two groups for the model fitting (training), and the remaining 10\% of each of them are used as testing samples.
This procedure is then repeated 100 times.
Because of the number of predictive variables is relatively large comparing to the subject size we have,
\emph{mgcv} and \emph{mgcv\_step} methods cannot converge in a reasonable computing time.
Hence, we do not include the results of these two algorithms in this example.
Figures \ref{figure5} (a) and (b) are the box-plots of AUCs and the computational times of all other methods based on 100 runs, respectively.
In this example, all methods, including \emph{GLM} and \emph{GLM\_step}, perform well in terms of their AUC averages.
The {\emph GAMBoost} has the largest average AUC with the smallest variation among these methods.
The linear logistic models (with and without a stepwise variable selection option) have smaller means of AUCs and larger variances than others do.
%
Most of the algorithms are computationally efficient in this case, except \emph{backfitting\_step}.
This is due to the stepwise variable selection procedure in \emph{backfitting\_step}.
Without stepwise option, the \emph{backfitting} is as efficient as others.
The differences of the computational times of \emph{GLM}, \emph{GLM\_step} and \emph{backfitting} are not statistically significant.
In fact, the CPU time used in \emph{backfitting} is only slightly longer than that of \emph{GLM} and \emph{GLM\_step}.
Thus, from both the improvement of AUCs and the corresponding computational time view points,
\emph{backfitting} and \emph{GAMBoost} are very ``cost-effective'' in this case.
The advantage of the \emph{backfitting} and \emph{GAMBoost} can also be seen from the plot of the average ROC curves in Figure \ref{figure8}. In order to have a clearer picture, we only show the left-upper $[0, 0.5]\times[0.5, 1]$ corner of the unit square.
Each curve plotted here is based on 100 replications, and the vertical bars on each curve are
the corresponding 95\% point-wise confidence intervals at the given false positive rates.
It is clearly shown that the ROC curve of \emph{GAMBoost} is significantly higher than other ROC curves in this range of false positive rate,
and all ROC curves from fitting a general additive logistic model, with different fitting algorithms, are higher than those of GLM methods.
This result confirms that the recommended method is useful even when only a specific range of false positive rate is of interest.
(Note that the ROC curves of \emph{GLM} and \emph{GLM\_step} are very close and thus cannot be distinguished in this figure.)
\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{ROC_Parkinsons}
\caption{Average ROC curves with confidence bars are obtained by each fitting method from Parkinson's data set with 100 replications.}
\label{figure8}
\end{figure}
\begin{figure}
\includegraphics[width=\textwidth]{gam_fitting}
\caption{The plots of the estimated functions of 13 predictors selected in the \emph{backfitting\_step}, where $x$-axis is the value of each predictor, the solid line is the estimated functions and the dot-lines represents $\pm$ 2 standard-error curves obtained from the fitting algorithm.}
\label{figure4}
\end{figure}
Figure \ref{figure4} shows the plots of estimated smooth functions and their corresponding partial residual plots for the selected predictors obtained from the \emph{backfitting\_step} method.
Because of the additive model assumption, we can look at one variable at a time by keeping the others fixed.
Thus, the interpretation of the relations between predictors and response variable is similar to that in a linear logistic model \citep[see][]{GAM}.
From this figure, we can see the impacts of the individual variables on the odds-ratio.
For example, the left-upper corner is a plot of the curve of the estimated association function for MDVP.Fo.Hz,
and it shows the functional relation between the log odds-ratio (response variable) and this variable.
When the values of all other variables are fixed, this plot shows that MDVP.Fo.Hz and the response is positively associated at the beginning and then turns to negatively associated as the value of the variable becomes large. Compared to that in the association plot for MDVP.Flo.Hz at the first row and second column, it clearly shows that these two variables have different associations with the responses. Because the curve of MDVP.Flo.Hz stays near 0, it indicates that the association of MDVP.Flo.Hz with the responses is not as strong as MDVP.Fo.Hz.
In addition, the picture at the second row and forth column shows that the estimated function of variable {\it MDVP.APQ} is almost flat, and the $\pm 2$ standard deviation curves of it cover $0$ at most values of {\it MDVP.APQ} in this picture, except for the values around 0.02. This suggests that this variable might have just a little impact on the (log) odds-ratio \citep[see][]{marra-wood12, binder-tutz08}.
(Please note that we just use this as an example for illustration purposes and no disease-related inference is made here.)
Thus, we can clearly find the impacts of individual variables on the responses through these plots
as well as the certain levels of the interpretation ability of variables.
It is clear that we cannot obtain this kind of nonlinear relation using a linear logistic model.
The way of interpretation is similar to that of a linear logistic model except that they are in functional form instead of slopes.
More discussions about the interpretation of GAMs can be easily found in the literature such as \citet{tutz2006generalized,Wood06}
and textbooks about GAM as \citet{marx1998direct, GAM, hastie-tibshirani-friedman2001}.
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{CTG_figure}
\caption{(a) AUC and (b) computational cost time measured by each fitting method with 10 resampling data from cardiotocography data set if label ``suspect'' belongs to ``normal.'' (c) AUC and (d) computational cost time measured by each fitting method with 10 resampling data from cardiotocography data set if label ``suspect'' belongs to ``pathologic.''}
\label{figure7}
\end{figure}
\paragraph{Cardiotocography data set}
The cardiotocography data set used here consists of 2126 fetal cardiotocograms (CTGs), where 19 diagnostic features were measured, including fetal heart rate and uterine contraction features.
Based on the judgements of some expert obstetricians, these 2126 subjects are classified as three fetal states: normal, suspect, and pathologic.
Because we consider only two-class classification problems in this work,
we combine two neighbored classes into one class.
Thus, they becomes two binary classification problems:
(1) \{{\bf normal and suspect}\} versus \{{\bf pathologic}\} and
(2) \{{\bf normal}\} versus \{{\bf suspect and pathologic}\}.
For demonstration, they are treated separately as two different binary classification problems.
Besides the contents, a major difference between these two real data examples is that the cardiotocography data set has more observations and less predictors, which will affect the computational efficiency of some algorithms.
The results reported here are based on a smilier re-sampling plan as before.
Both \emph{mgcv} and \emph{mgcv\_step} algorithms will now converge in a reasonable time, so their results are included. In this example, we omit the plots of the estimated functions and only focus on AUCs and related model fitting issues that are useful for practitioners to choose suitable methods for their own studies.
Figures \ref{figure7} (a) and (c) are the box-plots of AUCs for the cases of \{normal and suspect\} versus \{pathologic\} and \{normal\} versus \{suspect and pathologic\}, respectively. Figures \ref{figure7} (b) and (d) show their corresponding computational times of all methods.
Although \emph{mgcv} and \emph{mgcv\_step} converge in a reasonable time in this example, they are still not as stable as other GAM-based counterparts are in terms of variation of AUCs.
The \emph{GAMBoost} algorithm takes a much longer time than others in this case.
(Note that in our study, \emph{GAMBoost} takes more than eight hours for one run, while the \emph{backfitting} and \emph{backfitting\_step} take only less than five minutes.)
This phenomenon is due to the large sample size of the cardiotocography data set.
This result is consistent with our simulations and the comments reported in the literature.
Besides \emph{mgcv} and \emph{mgcv\_step}, all other GAM-based fitting algorithms perform better than \emph{GLM} and \emph{GLM\_step} do in terms of their means of AUCs and variances.
%
Hence, when the sample size is large and the number of predictive variables is relatively small,
the \emph{backfitting} and \emph{backfitting\_step} algorithms are recommended.
Figure \ref{figure9} shows the ROC curves at the left-upper corner of the unit square, and Figure \ref{figure9} (a) is \{normal and suspect\} versus \{pathologic\} case, and \ref{figure9} (b) is for \{normal\} versus \{suspect and pathologic\} case.
From Figure \ref{figure9} (a), we can see that the ROC curves of \emph{Backfittting}, \emph{Backfitting\_step} and \emph{GAMBoost} are three methods on the top; meanwhile,
then the two curves of \emph{GLM}-based methods, and the curves of \emph{mvcv}-based methods are the worst two in this case.
%
Figure \ref{figure9} (b) shows that ROC curves of the \{normal\} versus \{suspect and pathologic\} case.
Here, the performances of all methods are very close.
However, we still find that all \emph{GAM}-based fitting algorithms are significantly better than the two \emph{GLM}-based methods in the low false positive rate range (i.e., less than 0.15 in this plot).
In fact, in both binary classification cases of this data set, using a general additive logistic model increases the sensitivity in the range of the low false positive rate.
This range is usually important when a high-specificity diagnostic rule is preferred. Hence, this is also an advantage of using the recommended method.
\begin{figure}
\centering
\includegraphics[width=0.9\textwidth]{ROC_CTG}
\caption{Average ROC curves with 95\% confidence bars of each fitting method based on 100 resampling replications using cardiotocography data set for (a) \{{\bf normal and suspect}\} versus \{{\bf pathologic}\} and (b) \{{\bf normal}\} versus \{{\bf suspect and pathologic}\}.}
\label{figure9}
\end{figure}
%\vspace{-1.cm}
\section{Summary}
In this paper, the conventional logistic classification function can be further improved in terms of AUC. We show that both numerically and asymptotically, the spline-based generalized additive logistic model will have a larger AUC than a conventional linear logistic model does. We also demonstrate that the classification performance of a conventional linear logistic regression model can be easily improved with a general additive logistic model without sacrificing much of its ease of interpretation using numerical results using some synthesized data sets and two real data sets.
Several numerical algorithms are used to fit a GAM, and their pros and cons are discussed based on the numerical results in our studies.
We found that the \emph{GAMBoost} method outperforms other methods when with a moderate sample size and variable length.
When the sample size becomes large, the \emph{backfitting} algorithms are more computationally efficient compared to the CPU times used for fitting a conventional linear logistic model.
It is worth mentioning that the most of the algorithms used here are ``parallel-computing friendly.'' (Please see their manual for further details).
That is, these algorithms can be used in some parallel computing environment, such as a PC-cluster or just a multi-core CPU, which is now very popular in desktop or portable computers, with little programing effort.
Hence, the proposed method is useful for improving the diagnostic performance without diminishing the ability of the interpretation of the final prediction model,
and it is ``cost-effective'' in terms of the computational time and the gain in AUC.
Moreover, the ROC curve of the additive logistic model is higher than that of its linear logistic model counterpart in the whole range of specificity. It implies that the partial area under ROC curve in the high-specificity range is always improved, which is an important and useful feature in many medical diagnosis situations, and it is especially beneficial for applications where the high specificity is a must.
%
%
%
%
\section*{Appendix A: Algorithms for fitting general additive models}\label{Methodology}
\renewcommand{\theequation}{A.\arabic{equation}}
%\setcounter{equation}{0}
\renewcommand{\thesection}{\Alph{subsection}}
\setcounter{section}{0}%\subsection{Fitting algorithms and numerical results}
\paragraph{Backfitting algorithm}
The backfitting algorithm employs a concept of conditional expectation; that is, if GAM is correct, then for any $k$,
%\begin{displaymath}
$E[Y-\alpha-\sum_{j\neq k}{f_j(X_j)}|X_k]=f_k(X_k).$
%\end{displaymath}
Following \eqref{gam_formula}, the backfitting algorithm is stated below;
\begin{enumerate}
\item[(i)] Initial step: $\hat{\alpha}={N^{-1}}\sum_1^N{y_i}, \hat{f}_j\equiv0$ for $j=1,\ldots,p$.
\item[(ii)] Cycle: $j=1,2,\ldots,p,\ldots,1,2,\ldots,p,\ldots$,
\begin{flalign*}
&\hat{f}_j\leftarrow S_j[\{y_i-\hat{\alpha}-\sum_{k\neq j}{\hat{f}_k(x_{ik})\}}_1^N],\\
&\hat{f}\leftarrow \hat{f}-\frac{1}{N}\sum_{i=1}^N{\hat{f}_j(x_{ij})},
\end{flalign*}
where $S_j(\cdot)$ denotes a smoothing spline for a target against the predictor $x_j$.
\item[(iii)] Continue Step (ii) until individual functions do not change.
\end{enumerate}
For a generalized additive logistic model as in \eqref{gam_logistic_formula}, Hastie and Tibshirani proposed a local-scoring algorithm (see Chapter 4 of their book),
which is a maximum likelihood approach under a linear logistic regression model setup and a Newton-Raphson iterative type of method. The local-scoring algorithm has the following steps:
\begin{enumerate}
\item[(i)] Initial step: $\hat{\alpha}=\log\left(\bar{y}/(1-\bar{y})\right), \hat{f}_j\equiv0$ for $j=1,\ldots,p$.
\item[(ii)] Cycle:
\begin{enumerate}
\item[(a)] Construct a new response:
\begin{displaymath}
z_i=\hat{\alpha}+\sum_{j=1}^p{\hat{f}_j(x_{ij})}+\frac{y_i-\hat{p}_i}{\hat{p}_i(1-\hat{p}_i)},
\end{displaymath}
where $\hat{p}_i=\exp\{\hat{\alpha}+\sum{\hat{f}_j(x_{ij})\}/[1+\hat{\alpha}+\exp\{\sum{\hat{f}_j(x_{ij}) }\}}].$
\item[(b)] Construct weights: $w_i=\hat{p}(1-{\hat{p}})$.
\item[(c)]Estimate new $\hat{\alpha}$ and $\hat{f}_j$ for any $j$ by fitting a weighted additive model to $z_i$ with weights $w_i$ via a backfitting algorithm.
\end{enumerate}
\item[(iii)] Continue Step (ii) until the individual functions do not change.
\end{enumerate}
(Note that the GAM package in R can perform both the backfitting and local scoring algorithms.)
\paragraph{Simultaneous estimation of all components with optimization in a smoothing parameter space}
Both \eqref{gam_formula} and \eqref{gam_logistic_formula} can be modeled using the $B$-spline method:
$
f_j(x)=\sum_{i=1}^K{\beta_{ij}B_{i,m}(x)},
$
where $B_{i,m}$ is the $i$th $B$-spline basis function of order $m$ for the $K+m+2$ evenly spaced knots,
and $\beta_{ij}$ is an unknown coefficient of $B_{i,m}$ for $f_j$.
%
To prevent overfitting and have a better prediction, Marx and Eilers replaced the basic $B$-splines with a penalized $B$-splines, called $P$-splines \citep[see][]{marx1998direct}.
Thus, when estimating the unknown $\beta_{ij}$ by maximizing a likelihood function,
the $P$-splines will attach a different penalty on the adjacent coefficients of the $B$-splines to guarantee their smoothness.
Therefore, to fit a GAM becomes to maximize the penalized likelihood below:
\begin{equation}\label{Penalized-likelihood}
l^{\star}=l(\mathbf{y};\boldsymbol{\beta})-\frac{1}{2}\sum_{j=1}^p{\lambda_j\boldsymbol{\beta}_j^T\mathbf{P}\boldsymbol{\beta}_j},
\end{equation}
where $\lambda$, for all $j$, are smoothing parameters, $\mathbf{P}$ is the penalty matrix,
and $l(\mathbf{y};\boldsymbol{\beta})$ is the log-likelihood of GAM.
For a generalized additive logistic model as in \eqref{gam_logistic_formula}, the log-likelihood with a penalized first order differences becomes
\begin{displaymath}
l^{\star}=l(\mathbf{y};\boldsymbol{\beta})-\frac{1}{2}\sum_{j=1}^p{\left\{\lambda_j\sum_{i=1}^{K-1}{(\beta_{i+1,j}-\beta_{i,j})^2}\right\}},
\end{displaymath}
where $l(\mathbf{y};\boldsymbol{\beta})=\sum_{i=1}^n{\{y_i\log{p_i}+(1-y_i)\log{(1-p_i)}\}}$.
\citet{wood2004stable} provided a stable and efficient approach to estimate all functions, simultaneously. This feature is available in the \emph{mgcv} package in R.
\paragraph{Likelihood-based boosting method}
%Boosting is a powerful technology and has been successfully derived for improving classification problems.
Its basic idea of the boosting algorithm is to combine some ``weaker classifiers'' to produce a more powerful classifier
that can reduce misclassification error using an iterative adaptive weighting scheme on observations.
\citet{tutz2006generalized} use it as a way of fitting an additive expansion in a set of elementary basis functions by minimizing a loss function, such as a likelihood-based loss function.
They proposed the GAMBoost procedure to fit GAM with this boosting technology via maximizing the log-likelihood \eqref{Penalized-likelihood}.
In the statements below, we also use similar notations as that in the backfitting.
The GAMBoost algorithm for fitting a generalized additive logistic model \eqref{gam_logistic_formula} is stated as follows:
\begin{enumerate}
\item[(i)] Definition: $\eta_i=\alpha+f_1(x_{i1})+\ldots+f(x_{in})$, $\mu_i=h(\eta_i)={\exp(\eta_i)}/{[1+\exp(\eta_i)]}$.
\item[(ii)] Initialize: $\hat{\alpha}=\log\left(\bar{y}/(1-\bar{y})\right), \hat{f}_j=\hat{f}^{(0)}_j\equiv0$ for $j=1,\ldots,p$, yielding $\hat{\boldsymbol{\eta}}^{(0)T}=(\hat{\eta}^{(0)}_1,\ldots,\hat{\eta}^{(0)}_n)=(\hat{\alpha},\ldots,\hat{\alpha})$, $\hat{\boldsymbol{\mu}}^{(0)}=h(\hat{\boldsymbol{\eta}}^{(0)})$.
\item[(iii)] Cycle: $k=1,2,\ldots$,
\begin{enumerate}
\item[] (a) Estimate all components: For $j=1,\ldots,p$ compute
\begin{displaymath}
% % % % check the following formula
f_{j,\text{new}}=\mathbf{B}_j(\mathbf{B}_j^T\hat{\mathbf{W}}\mathbf{B}+\boldsymbol{\lambda}\mathbf{P})^{-1}(\mathbf{B}_j^T\hat{\mathbf{W}}\hat{\mathbf{D}}^{-1}(\mathbf{y}-\hat{\boldsymbol{\mu}})),
\end{displaymath}
where
$\mathbf{B}^T_j=(\mathbf{B}_{1j}, \ldots, \mathbf{B}_{nj})$,
$\mathbf{B}_{ij}=({B_{1,m}(x_{ij}), \ldots, B_{K,m}(x_{ij})})$,\\
$\hat{\mathbf{D}}=\text{diag}(h'(\hat{\eta}^{(k)}_1), \ldots, h'(\hat{\eta}^{(k)}_n))$,
$\hat{\mathbf{W}}=\hat{\mathbf{D}}\Sigma\hat{\mathbf{D}}$,
and
$\Sigma=\text{diag}(\text{var}(y_1),\ldots,\text{var}(y_1))$.
\item[] (b) Selection: Set $f_j=f^{(l)}_j+f_{j,\text{new}}$ yielding $\hat{\boldsymbol{\eta}}_{j,\text{new}}$ and\\ get $s=\arg{\max_j{\{\text{Dev}(\hat{\boldsymbol{\eta}}^{(k)})-\text{Dev}(\hat{\boldsymbol{\eta}}_{j,\text{new}})\}}}$, where $\text{Dev}(\boldsymbol{\eta})$ denotes the deviance when $\boldsymbol{\eta}$ is the fitted predictor.
\item[] (c) Update: Set $\hat{f}^{(k+1)}_{j+1}=\hat{f}^{(k)}_j+f_{s,\rm{new}}$.
\end{enumerate}
\end{enumerate}
The R package, {GAMBoost} is developed based on this algorithm.
%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix
\section*{References}
%% \label{}
%% If you have bibdatabase file and want bibtex to generate the
%% bibitems, please use
%%
\bibliographystyle{elsarticle-harv}
%% \bibliography{<your bibdatabase>}
\begin{thebibliography}{19}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\expandafter\ifx\csname url\endcsname\relax
\def\url#1{\texttt{#1}}\fi
\expandafter\ifx\csname urlprefix\endcsname\relax\def\urlprefix{URL }\fi
\bibitem[{Bache and Lichman(2013)}]{Bache+Lichman:2013}
Bache, K., Lichman, M., 2013. {UCI} machine learning repository.
\newline\urlprefix\url{http://archive.ics.uci.edu/ml}
\bibitem[{Binder and Tutz(2008)}]{binder-tutz08}
Binder, H., Tutz, G., 2008. A comparison of methods for the fitting of
generalized additive models. Stat. Comput 18, 87 -- 99.
\bibitem[{Cristianini and Shawe-Taylor(2000)}]{SVM-intro}
Cristianini, N., Shawe-Taylor, J., 2000. An Introduction to Support Vector
Machines and Other Kernel-based Learning Methods. Cambridge University Press,
The Edinburgh Building, Cambridge CB2 8RU, UK.
\bibitem[{Eguchi and Copas(2002)}]{eguchi2002class}
Eguchi, S., Copas, J., 2002. A class of logistic-type discriminant functions.
Biometrika 89~(1), 1--22.
\bibitem[{Friedman et~al.(2000)Friedman, Hastie, and Tibshirani}]{Friedman2k}
Friedman, J., Hastie, T., Tibshirani, R., 2000. Additive logistic regression: a
statistical view of boosting. Ann. Statist., 337 -- 407.
\bibitem[{Hastie and Tibshirani(1986)}]{hastie}
Hastie, T., Tibshirani, R., 1986. Generalized additive models. Statistical
Science 1~(3), 297 -- 318.
\bibitem[{Hastie and Tibshirani(1990)}]{GAM}
Hastie, T., Tibshirani, R., 1990. Generalized additive models. Vol.~43. CRC
Press.
\bibitem[{Hastie et~al.(2001)Hastie, Tibshirani, and
Friedman}]{hastie-tibshirani-friedman2001}
Hastie, T., Tibshirani, R., Friedman, J., 2001. The Elements of Statistical
Learning. Springer, Berlin.
\bibitem[{Little et~al.(2007)Little, McSharry, Roberts, Costello, Moroz,
et~al.}]{little2007exploiting}
Little, M.~A., McSharry, P.~E., Roberts, S.~J., Costello, D.~A., Moroz, I.~M.,
et~al., 2007. Exploiting nonlinear recurrence and fractal scaling properties
for voice disorder detection. BioMedical Engineering OnLine 6~(1), 23.
\bibitem[{Marra and Wood(2012)}]{marra-wood12}
Marra, G., Wood, S., 2012. Coverage properties of confidence intervals for
generalized additive model components. Scandinavian Journal of Statistics
39~(1), 53 --74.
\bibitem[{Marx and Eilers(1998)}]{marx1998direct}
Marx, B.~D., Eilers, P.~H., 1998. Direct generalized additive modeling with
penalized likelihood. Computational Statistics \& Data Analysis 28~(2),
193--209.
\bibitem[{Minka(2007)}]{Minka07}
Minka, T., March 26 2007. A comparison of numerical optimizers for logistic
regression.
\newline\urlprefix\url{http://research.microsoft.com/en-us/um/people/minka/papers/logreg/}
\bibitem[{Pepe(2004)}]{pepe2004statistical}
Pepe, M.~S., 2004. The statistical evaluation of medical tests for
classification and prediction. Oxford University Press, USA.
\bibitem[{Stone(1985)}]{stone1985additive}
Stone, C.~J., 1985. Additive regression and other nonparametric models. The
annals of Statistics, 689--705.
\bibitem[{Tian et~al.(2007)Tian, Cai, Goetghebeur, and Wei}]{Tian07}
Tian, L., Cai, T., Goetghebeur, E., Wei, L.~J., 2007. Model evaluation based on
the sampling distribution of estimated absolute prediction error. Biometrika
94~(2), 297--311.
\bibitem[{Tutz and Binder(2006)}]{tutz2006generalized}
Tutz, G., Binder, H., 2006. Generalized additive modeling with implicit
variable selection by likelihood-based boosting. Biometrics 62~(4), 961--971.
\bibitem[{Wood(2004)}]{wood2004stable}
Wood, S., 2004. Stable and efficient multiple smoothing parameter estimation
for generalized additive models. Journal of the American Statistical
Association 99~(467).
\bibitem[{Wood(2006)}]{Wood06}
Wood, S., 2006. Generalized Additive Models: An Introduction with R. Chapman \&
Hall.
\bibitem[{Wood(2008)}]{Wood08}
Wood, S., 2008. Fast stable direct fitting and smoothness selectionfor
generalized additive models. Journal of Royal Statist. Soc. B. 70~(3), 495 --
518.
\end{thebibliography}
%% else use the following coding to input the bibitems directly in the
%% TeX file.
%\bibliography{bib}
%\begin{thebibliography}{00}
%% \bibitem[Author(year)]{label}
%% Text of bibliographic item
%\bibitem[ ()]{}
%\end{thebibliography}
\end{document}
\endinput
%%
%% End of file `elsarticle-template-harv.tex'.