Consistent Individualized Feature Attribution for Tree Ensembles

Extracted source text · arXiv 1802.03888

\def\year{2018}\relax
%File: formatting-instruction.tex
\documentclass[letterpaper]{article} %DO NOT CHANGE THIS
\usepackage{aaai18}  %Required
\usepackage{times}  %Required
\usepackage{helvet}  %Required
\usepackage{courier}  %Required
\usepackage{url}  %Required
\usepackage{graphicx}  %Required
\frenchspacing  %Required
\setlength{\pdfpagewidth}{8.5in}  %Required
\setlength{\pdfpageheight}{11in}  %Required
%PDF Info Is Required:
  \pdfinfo{
/Title (2018 Formatting Instructions for Authors Using LaTeX)
/Author (AAAI Press Staff)}
\setcounter{secnumdepth}{0}  
 \begin{document}
% The file aaai.sty is the style file for AAAI Press 
% proceedings, working notes, and technical reports.
%
\title{2018 Formatting Instructions \\for Authors Using \LaTeX{}}
\author{AAAI Press\\
Association for the Advancement of Artificial Intelligence\\
2275 East Bayshore Road, Suite 160\\
Palo Alto, California 94303\\
}
\maketitle
\begin{abstract}
AAAI creates proceedings, working notes, and technical reports directly from electronic source furnished by the authors. To ensure that all papers in the publication have a uniform appearance, authors must adhere to the following instructions. 
\end{abstract}

\noindent Congratulations on having a paper selected for inclusion in an AAAI Press proceedings or technical report! This document details the requirements necessary to get your accepted paper published using \LaTeX{}. If you are using Microsoft Word, instructions are provided in a different document. If you want to use some other formatting software, you must obtain permission from AAAI Press first. 

The instructions herein are provided as a general guide for experienced \LaTeX{} users. If you do not know how to use \LaTeX{}, do not use it to format your paper. AAAI cannot provide you with support and the accompanying style files are \textbf{not} guaranteed to work. If the results you obtain are not in accordance with the specifications you received, you must correct your source file to achieve the correct result. 

These instructions are generic. Consequently, they do not include specific dates, page charges, and so forth. Please consult your specific written conference instructions for details regarding your submission. Please review the entire document for specific instructions that might apply to your particular situation. All authors must comply with the following:

\begin{itemize}
\item You must use the 2018 AAAI Press \LaTeX{} style file and bib file, which are located in the 2018 author kit.
\item You must complete, sign, and return by the deadline the AAAI copyright form (proceedings authors) or distribution license (technical report authors).
\item You must read and format your paper source and PDF according to the formatting instructions for authors.
\item You must submit your electronic files and abstract using our electronic submission form \textbf{on time.}
\item You must pay any required page or formatting charges to AAAI Press so that they are received by the deadline.
\item You must check your paper before submitting it, ensuring that it compiles without error, and complies with the guidelines found in the author kit.
\end{itemize}

\section{Copyright}
All papers submitted for publication by AAAI Press must be accompanied by a valid signed copyright form or, in the case of technical reports, by a valid signed permission to distribute form. There are no exceptions to this requirement. You must send us the original version of this form. However, to meet the deadline, you may fax (1-650-321-4457) or scan and e-mail the form (pubforms18@aaai.org) to AAAI by the submission deadline, and then mail the original via postal mail to the AAAI office. \textbf{If you fail to send in a signed copyright or permission form, we will be unable to publish your paper. There are no exceptions to this policy.} You will find PDF versions of the AAAI copyright and permission to distribute forms in the author kit.

\section{Formatting Requirements in Brief}
We need source and PDF files that can be used in a variety of ways and can be output on a variety of devices. The design and appearance of the paper is governed by the aaai style file. 
\textbf{You must not make any changes to the aaai style file, nor use any commands, packages, style files, or macros within your own paper that alter that design, including, but not limited to spacing, floats, margins, fonts, font size, and appearance.} AAAI imposes  requirements on your source and PDF files that must be followed. Most of these requirements are based on our efforts to standardize conference manuscript properties and layout. All papers submitted to AAAI for publication will be recompiled for standardization purposes. Consequently, every paper submission must comply with the following requirements:

\begin{itemize}
\item Your .tex file must compile in PDF\LaTeX{} --- \textbf{ no .ps or .eps figure files.}
\item All fonts must be embedded in the PDF file --- \textbf{ this includes your figures.}
\item Modifications to the style file, whether directly or via commands in your document may not be made, most especially when made in an effort to avoid extra page charges or make your paper fit in a specific number of pages.
\item No type 3 fonts may be used (even in illustrations).
\item You may not alter the spacing above and below captions, figures, headings, and subheadings.
\item You may not alter the font sizes of text elements, footnotes, heading elements, captions, or title information (for references and tables and mathematics, please see the the limited exceptions provided herein).
\item You may not alter the line spacing of text.
\item Your title must follow Title Case capitalization rules (not sentence case).
\item Your .tex file must include completed metadata to pass-through to the PDF (see PDFINFO below)
\item \LaTeX{} documents must use the Times or Nimbus font package (do not use Computer Modern for the text of your paper).
\item No \LaTeX{} 209 documents may be used or submitted.
\item Your source must not require use of fonts for non-Roman alphabets within the text itself. If your paper includes symbols in other languages (such as, but not limited to, Arabic, Chinese, Hebrew, Japanese, Thai, Russian and other Cyrillic languages), you must restrict their use to bit-mapped figures.
\item Fonts that require non-English language support (CID and Identity-H) must be converted to outlines or 300 dpi bitmap or removed from the document (even if they are in a graphics file embedded in the document). 
\item Two-column format in AAAI style is required for all papers.
\item The paper size for final submission must be US letter without exception.
\item The source file must exactly match the PDF.
\item The document margins must be as specified in the formatting instructions.
\item The number of pages and the file size must be as specified for your event.
\item No document may be password protected.
\item Neither the PDFs nor the source may contain any embedded links or bookmarks.
\item Your source and PDF must not have any page numbers, footers, or headers.
\item Your PDF must be compatible with Acrobat 5 or higher.
\item Your \LaTeX{} source file (excluding references) must consist of a \textbf{single} file (use of the ``input" command is not allowed.
\item Your graphics must be sized appropriately outside of \LaTeX{} (do not use the ``clip" command) .
\end{itemize}

If you do not follow the above requirements, your submission will be subject to reformatting and special handling fees that can easily exceed the extra page fee.

\section{What Files to Submit}
You must submit the following items to ensure that your paper is published:
\begin{itemize}
\item A fully-compliant PDF file.
\item Your  \LaTeX{}  source file submitted as a \textbf{single} .tex file (do not use the ``input" command to include sections of your paper --- every section must be in the single source file). The only exception is the reference list, which you should include separately. Your source must compile on our system, which includes the standard \LaTeX{} support files.
\item Only the graphics files used in compiling paper.
\item The \LaTeX{}-generated files (e.g. .aux and .bib file, etc.) for your compiled source.
\item If you have used an old installation of \LaTeX{}, you should include algorithm style files). If in doubt, include it.
\end{itemize}

Your \LaTeX{} source will be reviewed and recompiled on our system (if it does not compile, you may incur late fees).   \textbf{Do not submit your source in multiple text files.} Your single \LaTeX{} source file must include all your text, your bibliography (formatted using aaai.bst), and any custom macros. Accompanying this source file, you must also supply any nonstandard (or older) referenced style files and all your referenced graphics files. 

Your files should work without any supporting files (other than the program itself) on any computer with a standard \LaTeX{} distribution. Place your PDF and source files in a single tar, zipped, gzipped, stuffed, or compressed archive. Name your source file with your last (family) name.

\textbf{Do not send files that are not actually used in the paper.} We don't want you to send us any files not needed for compiling your paper, including, for example, this instructions file, unused graphics files, standard style files, additional material sent for the purpose of the paper review, and so forth.

\textbf{Obsolete style files.}  The commands for some common packages (such as some used for algorithms), may have changed. Please be certain that you are not compiling your paper using old or obsolete style files. 

\section{Using \LaTeX{} to Format Your Paper}

The latest version of the AAAI style file is available on AAAI's website. Download this file and place it in  the \TeX\ search path. Placing it in the same directory as the paper should also work. You must download the latest version of the complete author kit so that you will have the latest instruction set and style file.

\subsection{Document Preamble}

In the \LaTeX{} source for your paper, you \textbf{must} place the following lines as shown in the example in this subsection. This command set-up is for three authors. Add or subtract author and address lines as necessary, and uncomment the portions that apply to you. In most instances, this is all you need to do to format your paper in the Times font. The helvet package will cause Helvetica to be used for sans serif. These files are part of the PSNFSS2e package, which is freely available from many Internet sites (and is often part of a standard installation).

Leave the setcounter for section number depth commented out and set at 0 unless you want to add section numbers to your paper. If you do add section numbers, you must uncomment this line and change the number to 1 (for section numbers), or 2 (for section and subsection numbers). The style file will not work properly with numbering of subsubsections, so do not use a number higher than 2.

If (and only if) your author title information will not fit within the specified  height allowed, put \textbackslash setlength \textbackslash titlebox{2.5in} in your preamble. Increase the height until the height error disappears from your log. You may not use the  \textbackslash setlength command elsewhere in your paper, and it may not be used to reduce the height of the author-title box.

\subsubsection{The Following Must Appear in Your Preamble}
\begin{quote}
\begin{scriptsize}\begin{verbatim}
\documentclass[letterpaper]{article}
\usepackage{aaai}
\usepackage{times}
\usepackage{helvet}
\usepackage{courier}
\usepackage{url}
\usepackage{graphicx}
\frenchspacing
% Add additional packages here. The following
% packages may NOT be used  (this list
% is not exhaustive:
% authblk, caption, CJK, float, fullpage, geometry, 
%hyperref, layout, nameref, natbib, savetrees, 
%setspace, titlesec, tocbibind, ulem
%
%US Lettersize Paper Is Required
\setlength{pdfpagewidth}{8.5in}
\setlength{pdfpageheight}{11in}\\
%
%
% PDFINFO
% You are required to complete the following
% for pass-through to the PDF. 
% No LaTeX commands of any kind may be
% entered. The parentheses and spaces 
% are an integral part of the 
% pdfinfo script and must not be removed.
%
\pdfinfo{
/Title (Input Your Paper Title Here)
/Author (John Doe, Jane Doe)
/Keywords (Input your keywords in this optional area)
}
%
%Section Numbers
% Uncomment if you want to use section numbers
% and change the 0 to a 1 or 2
% \setcounter{secnumdepth}{0}

% Title and Author Information Must Immediately Follow
% the pdfinfo within the preamble
%
\title{Title}\\
\author\{Author 1 \ and Author 2\\
Address line\\
Address line\\
\ And\\
Author 3\\
Address line\\
Address line
}\\
%
\end{verbatim}\end{scriptsize}
\end{quote}

\subsection{Preparing Your Paper}

After the preamble above, you should prepare your paper as follows:
\begin{quote}
\begin{scriptsize}\begin{verbatim}
%
\begin{document}
\maketitle
\begin{abstract}
%...
\end{abstract}\end{verbatim}\end{scriptsize}
\end{quote}

\subsubsection{The Following Must Conclude Your Document}
\begin{quote}
\begin{scriptsize}\begin{verbatim}
%References and End of Paper
%These lines must be placed at the end of your paper
\bibliography{Bibliography-File}
\bibliographystyle{aaai}
\end{document}
\end{verbatim}\end{scriptsize}
\end{quote}

\subsection{Inserting Document Metadata with \LaTeX{}}
PDF files contain document summary information that enables us to create an Acrobat index (pdx) file, and also allows search engines to locate and present your paper more accurately. \textit{Document metadata for author and title are REQUIRED.} You may not apply any script or macro to implementation of the title, author, and metadata information in your paper.

\textit{Important:} Do not include \textit{any} \LaTeX{} code or nonascii characters (including accented characters) in the metadata. The data in the metadata must be completely plain ascii. It may not include slashes, accents, linebreaks, unicode, or any \LaTeX{} commands. Type the title exactly as it appears on the paper (minus all formatting). Input the author names in the order in which they appear on the paper (minus all accents), separating each author by a comma. You may also include keywords in the Keywords field.




\begin{quote}
\begin{scriptsize}\begin{verbatim}
\begin{document}\\
\maketitle\\
...\\
\bibliography{Bibliography-File}\\
\bibliographystyle{aaai}\\
\end{document}\\
\end{verbatim}\end{scriptsize}
\end{quote}


\subsection{Illegal Commands}
There are a number of packages, commands, scripts, and macros that are incompatable with aaai18.sty. The common ones are provided later on in this document (See some Common  Errors to Avoid).  Generally, if a command, package, script, or macro alters floats, margins, fonts, sizing, linespacing, or the presentation of the references and citations, it is unacceptable. The following commands are some of the more common ones that may not be used in your paper (this list is not exhaustive --- there are others):
\begin{itemize}
\item \textbackslash renewcommand (in almost all instances)
\item \textbackslash baselinestretch
\item \textbackslash setlength (except for titlebox)
\item \textbackslash input
\item \textbackslash vspace or vskip (when used before or after a section or subsection or figure or table)
\item \textbackslash addtolength 
\item \textbackslash columnsep
\item \textbackslash top margin (or text height or addsidemargin or even side margin)
\item trim or clip (used to crop figures)
\item any command that globally alters floats, space above and below figures and tables
\end{itemize}

\subsection{Removing Certain Commands in Your Final Paper}
For your final camera ready copy, you must not use any page break commands, including, but not limited to:
\begin{itemize}
\item \textbackslash newpage
\item \textbackslash break
\item \textbackslash clearpage
\item \textbackslash pagebreak
\end{itemize}
(References must flow directly after the text without breaks.) Note that this may \textit{not} the case when submitting a paper for review. Some conferences require references to be on a separate page during the review process. AAAI Press, however, does not require this condition for the final paper. You must also be certain that your \textbackslash pdfino author field includes all your authors. The latter step is often forgotten.


\subsection{Paper Size, Margins, and Column Width}
Papers must be formatted to print in two-column format on 8.5 x 11 inch US letter-sized paper. The margins must be exactly as follows: 
\begin{itemize}
\item Top margin: .75 inches
\item Left margin: .75 inches
\item Right margin: .75 inches
\item Bottom margin: 1.25 inches
\end{itemize} 


The default paper size in most installations of \LaTeX{} is A4. However, because we require that your electronic paper be formatted in US letter size, the preamble we have provided includes commands that alter the default to US letter size. Please note that using any other package to alter page size (such as, but not limited to the Geometry package) will result in your final paper being rejected. 


\subsubsection{Column Width and Margins.}
To ensure maximum readability, your paper must include two columns. Each column should be 3.3 inches wide (slightly more than 3.25 inches), with a .375 inch (.952 cm) gutter of white space between the two columns. The aaai18.sty file will automatically create these columns for you. 

\subsection{Overlength Papers}
If your paper is too long, turn on \textbackslash frenchspacing, which will reduce the space after periods. Next,  shrink the size of your graphics. Use \textbackslash centering instead of \textbackslash begin\{center\} in your figure environment. For mathematical environments, you may reduce fontsize {\bf but not below 6.5 point}. You may also alter the size of your bibliography by inserting \textbackslash fontsize\{9.5pt\}\{10.5pt\} \textbackslash selectfont
right before the bibliography (the minimum size is \textbackslash fontsize\{9.0pt\}\{10.0pt\}. 

Commands that alter page layout are forbidden. These include \textbackslash columnsep, \textbackslash topmargin, \textbackslash topskip, \textbackslash textheight, \textbackslash textwidth, \textbackslash oddsidemargin, and \textbackslash evensizemargin (this list is not exhaustive). If you alter page layout, you will be required to pay the page fee \textit{plus} a reformatting fee. Other commands that are questionable and may cause your paper to be rejected include  \textbackslash parindent, and \textbackslash parskip. Commands that alter the space between sections are forbidden. The title sec package is not allowed. Regardless of the above, if your paper is obviously ``squeezed" it is not going to to be accepted. Options for reducing the length of a paper include reducing the size of your graphics, cutting text, or paying the extra page charge (if it is offered). 

\subsection{Figures}
Your paper must compile in PDF\LaTeX{}. Consequently, all your figures must be .jpg, .png, or .pdf. You may not use  the .gif (the resolution is too low), .ps, or .eps file format for your figures.

When you include your figures, you must crop them \textbf{outside} of \LaTeX{}. The command \textbackslash includegraphics*[clip=true, viewport 0 0 10 10]{...} might result in a PDF that looks great, but the image is \textbf{not really cropped.} The full image can reappear (and obscure whatever it is overlapping) when page numbers are applied or color space is standardized. 

\subsection{Type Font and Size}
Your paper must be formatted in Times Roman or Nimbus. We will not accept papers formatted using Computer Modern or Palatino or some other font as the text or heading typeface. Sans serif, when used, should be Courier. Use Symbol or Lucida or Computer Modern for \textit{mathematics only. } 

Do not use type 3 fonts for any portion of your paper, including graphics. Type 3 bitmapped fonts are designed for fixed resolution printers. Most print at 300 dpi even if the printer resolution is 1200 dpi or higher. They also often cause high resolution imagesetter devices and our PDF indexing software to crash. Consequently, AAAI will not accept electronic files containing obsolete type 3 fonts. Files containing those fonts (even in graphics) will be rejected. 

Fortunately, there are effective workarounds that will prevent your file from embedding type 3 bitmapped fonts. The easiest workaround is to use the required times, helvet, and courier packages with \LaTeX{}2e. (Note that papers formatted in this way will still use Computer Modern for the mathematics. To make the math look good, you'll either have to use Symbol or Lucida, or you will need to install type 1 Computer Modern fonts --- for more on these fonts, see the section ``Obtaining Type 1 Computer Modern.")

If you are unsure if your paper contains type 3 fonts, view the PDF in Acrobat Reader. The Properties/Fonts window will display the font name, font type, and encoding properties of all the fonts in the document. If you are unsure if your graphics contain type 3 fonts (and they are PostScript or encapsulated PostScript documents), create PDF versions of them, and consult the properties window in Acrobat Reader. 

The default size for your type should be ten-point with twelve-point leading (line spacing). Start all pages (except the first) directly under the top margin. (See the next section for instructions on formatting the title page.) Indent ten points when beginning a new paragraph, unless the paragraph begins directly below a heading or subheading.

\subsubsection{Obtaining Type 1 Computer Modern for \LaTeX{}.}

If you use Computer Modern for the mathematics in your paper (you cannot use it for the text) you may need to download type 1 Computer fonts. They are available without charge from the American Mathematical Society:
http://www.ams.org/tex/type1-fonts.html. 

\subsection{Title and Authors}
Your title must appear in mixed case (nouns, pronouns, and verbs are capitalized) near the top of the first page, centered over both columns in sixteen-point bold type (twenty-four point leading). This style is called ``mixed case." Author's names should appear below the title of the paper, centered in twelve-point type (with fifteen point leading), along with affiliation(s) and complete address(es) (including electronic mail address if available) in nine-point roman type (the twelve point leading). (If the title is long, or you have many authors, you may reduce the specified point sizes by up to two points.) You should begin the two-column format when you come to the abstract. 

\subsubsection{Formatting Author Information}
Author information can be set in a number of different styles, depending on the number of authors and the number of affiliations you need to display. In formatting your author information, however, you may not use a table nor may you employ the \textbackslash authorblk.sty package. For several authors from the same institution, use \textbackslash and:

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{Author 1 \and ... \and Author n\\ 
Address line \\ ... \\ Address line}
\end{verbatim}\end{scriptsize}\end{quote}

\noindent If the names do not fit well on one line use:

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{Author 1} ... \\
{\bf \Large Author  ...  Author}\\ 
Address line \\ ... \\ Address line
}
\end{verbatim}\end{scriptsize}\end{quote}

\noindent For authors from different institutions, use \textbackslash And:

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{Author 1\\ Address line \\ ... \\ Address line 
\And ... \And Author n\\
Address line\\ ... \\ Address line}
\end{verbatim}\end{scriptsize}\end{quote}

\noindent To start a separate ``row" of authors, use \textbackslash AND:

\noindent If the title and author information does not fit in the area allocated, place
\textbackslash setlength\textbackslash titlebox\{\textit{height}\}
after the \textbackslash documentclass line where \{\textit{height}\} is  2.5in or greater.

\subsubsection{Formatting Author Information --- Alternative Method}
If your paper has a large number of authors from different institutions, you may use the following alternative method for displaying the author information.

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{AuthorOne},\textsuperscript{1}
\author{AuthorTwo},\textsuperscript{2}
\author{AuthorThree},\textsuperscript{3}
\author{AuthorFour},\textsuperscript{4}
\author{AuthorFive}, \textsuperscript{5}\\
\textsuperscript{1}AffiliationOne}\\
\textsuperscript{2}AffiliationTwo}\\
\textsuperscript{3}AffiliationThree}\\
\textsuperscript{4}AffiliationFour}\\
\textsuperscript{5}AffiliationFive}\\
\{email, email\}@affiliation.com,
email@affiliation.com,
email@affiliation.com, 
email@affiliation.com
\end{verbatim}\end{scriptsize}\end{quote}


\subsection{\LaTeX{} Copyright Notice}
The copyright notice automatically appears if you use aaai.sty. If you are creating a technical report, it is not necessary to include this notice. You may disable the copyright line using the \verb+\+nocopyrightcommand. To change the entire text of the copyright slug, use:
\textbackslash copyrighttext \{\emph{text}\}.
Either of these must appear before \textbackslash maketitle. Please be advised, however, that \textit{if you disable or change the copyright line and transfer of copyright is required, your paper will not be published.}

\subsection{Credits}
Any credits to a sponsoring agency should appear in the acknowledgments section, unless the agency requires different placement. If it is necessary to include this information on the front page, use
\textbackslash thanks in either the \textbackslash author or \textbackslash title commands.
For example:
\begin{quote}
\begin{small}
\textbackslash title\{Very Important Results in AI\textbackslash thanks\{This work is
 supported by everybody.\}\}
\end{small}
\end{quote}
Multiple \textbackslash thanks commands can be given. Each will result in a separate footnote indication in the author or title with the corresponding text at the botton of the first column of the document. Note that the \textbackslash thanks command is fragile. You will need to use \textbackslash protect.

Please do not include \textbackslash pubnote commands in your document.

\subsection{Abstract}
Follow the example commands in this document for creation of your abstract. Further indentation is not required. {Do not include references in your abstract!}

\subsection{Page Numbers}

Do not \textbf{ever} print any page numbers on your paper. 

\subsection{Text }
The main body of the paper must be formatted in ten-point with twelve-point leading (line spacing).

\subsection{Citations}
Citations within the text should include the author's last name and year, for example (Newell 1980). Append lower-case letters to the year in cases of ambiguity. Multiple authors should be treated as follows: (Feigenbaum and Engelmore 1988) or (Ford, Hayes, and Glymour 1992). In the case of four or more authors, list only the first author, followed by et al. (Ford et al. 1997).

\subsection{Extracts}
Long quotations and extracts should be indented ten points from the left and right margins. 

\begin{quote}
This is an example of an extract or quotation. Note the indent on both sides. Quotation marks are not necessary if you offset the text in a block like this, and properly identify and cite the quotation in the text. 

\end{quote}

\subsection{Footnotes}
Avoid footnotes as much as possible; they interrupt the reading of the text. When essential, they should be consecutively numbered throughout with superscript Arabic numbers. Footnotes should appear at the bottom of the page, separated from the text by a blank line space and a thin, half-point rule. 

\subsection{Headings and Sections}
When necessary, headings should be used to separate major sections of your paper. Remember, you are writing a short paper, not a lengthy book! An overabundance of headings will tend to make your paper look more like an outline than a paper. The aaai.sty package will create headings for you. Do not alter their size nor their spacing above or below. 

\subsubsection{Section Numbers}
The use of section numbers in AAAI Press papers is optional. To use section numbers in \LaTeX{}, uncomment the setcounter line in your document preamble and change the 0 to a 1 or 2. Section numbers should not be used in short poster papers.

\subsubsection{Section Headings.}
Sections should be arranged and headed as follows: 

\subsubsection{Acknowledgments.}
The acknowledgments section, if included, appears after the main body of text and is headed ``Acknowledgments." This section includes acknowledgments of help from associates and colleagues, credits to sponsoring agencies, financial support, and permission to publish. Please acknowledge other contributors, grant support, and so forth, in this section. Do not put acknowledgments in a footnote on the first page. If your grant agency requires acknowledgment of the grant on page 1, limit the footnote to the required statement, and put the remaining acknowledgments at the back. Please try to limit acknowledgments to no more than three sentences. 

\subsubsection{Appendices.}
Any appendices follow the acknowledgments, if included, or after the main body of text if no acknowledgments appear. 

\subsubsection{References}
The references section should be labeled ``References" and should appear at the very end of the paper (don't end the paper with references, and then put a figure by itself on the last page). A sample list of references is given later on in these instructions. Please use a consistent format for references. Poorly prepared or sloppy references reflect badly on the quality of your paper and your research. Please prepare complete and accurate citations.

\subsection{Illustrations and Figures}
Figures, drawings, tables, and photographs should be placed throughout the paper near the place where they are first discussed. Do not group them together at the end of the paper. If placed at the top or bottom of the paper, illustrations may run across both columns. Figures must not invade the top, bottom, or side margin areas. Figures must be inserted using the \textbackslash usepackage\{graphicx\}. Number figures sequentially, for example, figure 1, and so on. 

The illustration number and caption should appear under the illustration. Labels, and other text with the actual illustration must be at least nine-point type. 

If your paper includes illustrations that are not compatible with PDF\TeX{} (such as .eps or .ps documents), you will need to convert them. The epstopdf package will usually work for eps files. You will need to convert your ps files to PDF however.



\subsubsection{Low-Resolution Bitmaps.}
You may not use low-resolution (such as 72 dpi) screen-dumps and GIF files---these files contain so few pixels that they are always blurry, and illegible when printed. If they are color, they will become an indecipherable mess when converted to black and white. This is always the case with gif files, which should never be used. The resolution of screen dumps can be increased by reducing the print size of the original file while retaining the same number of pixels. You can also enlarge files by manipulating them in software such as PhotoShop. Your figures should be 300 dpi when incorporated into your document.

\subsubsection{\LaTeX{} Overflow.}
\LaTeX{} users please beware: \LaTeX{} will sometimes put portions of the figure or table or an equation in the margin. If this happens, you need to scale the figure or table down, or reformat the equation.{ \bf Check your log file!} You must fix any overflow into the margin (that means no overfull boxes in \LaTeX{}). If you don't, the overflow text will simply be eliminated. \textbf{Nothing is permitted to intrude into the margin or gutter.}

\subsubsection{Using Color.}
Use of color is restricted to figures only. It may never be used for any portion of the text of your paper. The archival version of your paper will be printed in black and white and grayscale. Consequently, because conversion to grayscale can cause undesirable effects (red changes to black, yellow can disappear, and so forth), we strongly suggest you avoid placing color figures in your document. If you do include color figures, you must (1) use the CMYK (not RGB) colorspace and (2)  be mindful of readers who may happen to have a color deficiency. Your paper must be decipherable without using color for distinction. 

\subsubsection{Drawings.}
We suggest you use computer drawing software (such as Adobe Illustrator or, (if unavoidable), the drawing tools in Microsoft Word) to create your illustrations. Do not use Microsoft Publisher. These illustrations will look best if all line widths are uniform (half- to two-point in size), and you do not create labels over shaded areas. Shading should be 133 lines per inch if possible. Use Times Roman or Helvetica for all figure call-outs. \textbf{Do not use hairline width lines} --- be sure that the stroke width of all lines is at least .5 pt. Zero point lines will print on a laser printer, but will completely disappear on the high-resolution devices used by our printers.

\subsubsection{Photographs and Images.}
Photographs and other images should be in grayscale (color photographs will not reproduce well; for example, red tones will reproduce as black, yellow may turn to white, and so forth) and set to a minimum of 300 dpi. Do not prescreen images.

\subsubsection{Resizing Graphics.}
Resize your graphics \textbf{before} you include them with LaTeX. You may \textbf{not} use trim or clip options as part of your \textbackslash includegraphics command. Resize the media box of your PDF using a graphics program instead. 

\subsubsection{Fonts in Your Illustrations}
You must embed all fonts in your graphics before including them in your LaTeX document.

\subsection{References} 
The AAAI style includes a set of definitions for use in formatting references with BibTeX. These definitions make the bibliography style fairly close to the one specified below. To use these definitions, you also need the BibTeX style file ``aaai.bst," available in the author kit on the AAAI web site. Then, at the end of your paper but before \textbackslash end{document}, you need to put the following lines:

\begin{quote}
\begin{small}
\textbackslash bibliographystyle\{aaai\}
\textbackslash bibliography\{bibfile1,bibfile2,...\}
\end{small}
\end{quote}

Please note that you are required to use \textbackslash bibliographystyle\{aaai\} for your references. You may not use named, plain, apalike, acm, ieeetr, siam, chicago, or any other style. Use of natbib is also not acceptable.

The list of files in the \textbackslash  bibliography command should be the names of your BibTeX source files (that is, the .bib files referenced in your paper).

The following commands are available for your use in citing references:
\begin{quote}
\begin{small}
\textbackslash cite: Cites the given reference(s) with a full citation. This appears as ``(Author Year)'' for one reference, or ``(Author Year; Author Year)'' for multiple references.\\
\textbackslash shortcite: Cites the given reference(s) with just the year. This appears as ``(Year)'' for one reference, or ``(Year; Year)'' for multiple references.\\
\textbackslash citeauthor: Cites the given reference(s) with just the author name(s) and no parentheses.\\
\textbackslash citeyear: Cites the given reference(s) with just the date(s) and no parentheses.
\end{small}
\end{quote}

\textbf{Warning:} The aaai.sty file is incompatible with the hyperref and natbib packages. If you use either, your references will be garbled and your paper will not be published.

Formatted bibliographies should look like the following examples.

\smallskip \noindent \textit{Book with Multiple Authors}\\
Engelmore, R., and Morgan, A. eds. 1986. \textit{Blackboard Systems.} Reading, Mass.: Addison-Wesley.

\smallskip \noindent \textit{Journal Article}\\
Robinson, A. L. 1980a. New Ways to Make Microcircuits Smaller. \textit{Science} 208: 1019--1026.

\smallskip \noindent \textit{Magazine Article}\\
Hasling, D. W.; Clancey, W. J.; and Rennels, G. R. 1983. Strategic Explanations in Consultation. \textit{The International Journal of Man-Machine Studies} 20(1): 3--19.

\smallskip \noindent \textit{Proceedings Paper Published by a Society}\\
Clancey, W. J. 1983. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. In \textit{Proceedings of the Eighth International Joint Conference on Artificial Intelligence,} 556--560. Menlo Park, Calif.: International Joint Conferences on Artificial Intelligence, Inc.

\smallskip \noindent \textit{Proceedings Paper Published by a Press or Publisher}\\
Clancey, W. J. 1984. Classification Problem Solving. In \textit{Proceedings of the Fourth National Conference on Artificial Intelligence,} 49--54. Menlo Park, Calif.: AAAI Press. 

\smallskip \noindent \textit{University Technical Report}\\
Rice, J. 1986. Poligon: A System for Parallel Problem Solving, Technical Report, KSL-86-19, Dept. of Computer Science, Stanford Univ. 

\smallskip \noindent \textit{Dissertation or Thesis}\\
Clancey, W. J. 1979. Transfer of Rule-Based Expertise through a Tutorial Dialogue. Ph.D. diss., Dept. of Computer Science, Stanford Univ., Stanford, Calif.

\smallskip \noindent \textit{Forthcoming Publication}\\
Clancey, W. J. 1986. The Engineering of Qualitative Models. Forthcoming.


\section{Some Common Errors to Avoid}

The following list includes a number of the most common mistakes made by authors when formatting their paper and the reponses that have been sent to the author regarding them:

\begin{small}
\begin{quote}
PDFINFO\\
The following required elements are missing from the preamble to your LaTeX source (or are malformed):
\begin{small}\begin{verbatim}
\pdfinfo{ 
/Title () 
/Author () 
}
\end{verbatim}\end{small}



A4\\
The PDF you submitted is A4. All PDFs must be US Letter sized. Submissions that do not conform to this requirement cannot be published. 
\smallskip


ABOVEDISPLAY\\
You've used \textbackslash abovedisplay, \textbackslash belowdisplay, \textbackslash above caption, and \textbackslash belowcaption to alter aaai18.sty. These commands are illegal and must be stripped from your source.
\smallskip


ABSTRACT INDENTATION\\
Remove the \textbackslash begin\{quote\} and \textbackslash \{endquote\} around your abstract. It might buy you some space.
\smallskip


ASPECT RATIO OF FIGURES CHANGED\\
You may scale the height OR the width of your figures, but not both.
\smallskip


AUTHOR LIST MALFORMED (AUTHBLK)\\
The authblk.sty package cannot be used. 
\smallskip


AUTHOR LIST MALFORMED\\
You did not use the built-in commands in aaai18.sty to format your authors. You need to follow the formatting instructions for authors using LaTeX. 
\smallskip


AUTHOR LIST TOO WIDE\\
Your author list goes beyond the accepted page width. Please break the line and insert \{\textbackslash bf \textbackslash Large\} around the authors on the second line. The author and title information must be presented as specified in the author instructions.
\smallskip


AUTHOR AFFLIATIONS TOO WIDE\\
Your author's affiliations go beyond the accepted page width. Please break the line.
\smallskip


BASELINESTRETCH\\
You've used \textbackslash baselinestretch --- a command specifically prohibited in the author formatting instructions. Please remove it.
\smallskip


CAPTION STYLE ALTERED (CAPTION.STY)\\
You've altered the font and style of captions. Captions must be rendered as specified by aaai.sty. They cannot be altered by changing the font, fontsize, or position, whether manually or by using caption.sty. Such revisions must be removed.
\smallskip


CJK\\
The CJK package cannot be used. Restrict use of non-roman alphabets to figures and tables, which must be compiled and converted to PDF, then imported as a figure into LaTeX.
\smallskip


CLIP\\
You may not use the clip to import only a portion of a graphic. It only places a fragile layer that screens the remained of the figure from view, but imports the entire figure into the PDF. The layer is fragile, and often fails when flattened. The full content of all figures must be imported. In addition and for the same reason, you may not use vspace to reposition a figure. All figures must be completely cropped and adjusted outside of LaTeX before being imported. 
\smallskip


COLUMN WIDTH\\
Unfortunately, your paper contains material that exceeds the column width. Please review your LaTeX log file and correct all the overfull boxes.
\smallskip


COMPILE ERRORS\\
Unfortunately, the source file you submitted will not compile. We are getting undefined control sequence errors. There are either missing packages in your preamble, or missing definitions, or perhaps you are using an old or modified style. For your information, we are using a complete 2017 install of TeXLive, complete with all updates as of September 1. The packages listed in your preamble are all installed on all our systems. If you are using a style file that is not in that package, you will need to include it. Whatever the reason, we need you to correct your LaTeX and resubmit your source files. (If your paper contains graphics, be certain that the fonts are embedded.) 
\smallskip


EPSF / EPSFIG / PSFIG\\
The epsf, epsfig, and psfig packages are obsolete. Use the graphicx package instead.
\smallskip


EULER\\
The euler package is obsolte
\smallskip


FIGURES HAVE BEEN CROPPED WITH LATEX\\
You have used trim and clip commands in your \textbackslash includegraphics statement. Please crop your graphics appropriately using a standard graphics program (not Preview). The masks created by LaTeX are fragile, and disappear when PDFs are combined together. This will result in text in your paper becoming obscured by the fully restored graphic you have imported.
\smallskip


FIGURES MISSING\\
Unfortunately, the source file you submitted are incomplete. You did not include the graphics files. As a result, your paper will not compile. You must upload a new compressed archive that contains all the files required to compile your paper on a different computer and network. (If your paper contains graphics, be certain that the fonts are embedded.) 
\smallskip

FLOATS ALTERED\\
You've altered floats.  The \textbackslash renewcommand cannot be used. The float package cannot be used. Please remove them.
\smallskip


FOOTNOTE STYLE ALTERED\\
Footnotes must follow the built-in style.
\smallskip


FONTS (EMBEDDING)\\
Unfortunately, one or more of the fonts in your paper (most likely the figures) were not embedded in your PDF. This can cause hidden and silent changes to characters once the paper is published, especially when it is combined with other PDFs in the proceedings. If you need to know how to verify the fonts in your figures, please search for ``font embedding pdf." You will find a number of methods, one or more of which may be suitable for your operating system and available software. We recommend using Acrobat Reader to check document properties in your PDF, as it is standards-based software. Consequently, you must embed all the fonts in all your graphics files. If you do not know how to do this, change them to png or jpg. 
\smallskip


FONTS (LANGUAGE)\\
We cannot accept files that require custom installation of non-roman font sets, including CJK and arabic.
 \smallskip


FONTS (TYPE 3)\\
Unfortunately, one or more of the fonts in your paper (most likely the figures) are type 3 PDF. If in your figures, you must either change them to type 1 embedded fonts or change them to png or jpg. If the problem is in your text, you will need to switch the package you are using that is calling type 3 fonts to one that uses type 1 fonts. (Blackboard fonts are often the problem in these cases.)
\smallskip


FULL PAGE USED\\
You used \textbackslash usepackage\{full page\}, which is a package specifically disallowed. You must remove it.
\smallskip


GEOMETRY USED\\
You used \textbackslash usepackage\{geometry\}, which is a package specifically disallowed. You must remove it.
\smallskip


GRAPHICS\\
The graphics package is obsolete. Use the graphicx package instead.
\smallskip


HYPERREF\\
The hyperref package may never be used. If you are using it for URLs, use url.sty instead
\smallskip


INCOMPATIBLE SCRIPT OR MACRO\\
You've used an incompatible script (such as \textbackslash maltepaper.sty) to automate completion of the title and author information on your paper and in the metadata. Such scripts impede correction of your paper and often are incompatible with Acrobat (producing corrupted metadata fields). Scripts and macros cannot be used to automate population of metadata, author or title information, or insertion of figures or tables.
\smallskip


LAYOUT\\
The layout package cannot be used. Use url.sty instead.
\smallskip


LINESPREAD\\
\textbackslash linespread alters aaai.sty. This command (along with \textbackslash baselineshift and others that alter aaai.sty) cannot be used. You must remove it.
\smallskip


LMODERN\\
You used \textbackslash usepackage\{lmodern\}, which is a package specifically disallowed. You must remove it.
\smallskip


LAYOUT USED\\
You used \textbackslash usepackage\{layout\}, which is a package specifically disallowed. You must remove it.
\smallskip


MBOX USED FOR FIGURES\\
You cannot use mboxes to insert figures. They circumvent the spacing requirements for aaai.sty. Use subfigures instead.
\smallskip


MULTIPLE TEX FILES\\
You did not submit a single .tex file as specified in the instructions. We require this to avoid production errors and to facilitate debugging. The \textbackslash input command is not allowed in your source. Please combine all your tex files into a single .tex file (references can remain in .bbl or .bib file).
\smallskip


NAMEREF\\
The nameref package cannot be used.
\smallskip


NATBIB\\
Natbib is incompatible with aaai18.sty and aaai.bst. It must be removed. See References Malformed below.
\smallskip
\smallskip


PAGE BREAKS \\
You've included pagebreaks in your final submission. (They are not required before your references accepted final papers and must be removed).
\smallskip

PAGE NUMBERS\\
Your paper contains page numbers. You need to remove them.
\smallskip

PDFLaTeX NOT USED\\
Your source will not compile in PDFLaTeX. You must convert your eps graphics to PDF.
\smallskip


PDFCOMMENT\\
You used pdfcomment, which is a disallowed package. You must remove it.
\smallskip


PSTRICKS\\
You used pstricks, which is a package specifically disallowed. You must remove it. (Use Tikz instead). If you are simply adding labels, use pinlabel.sty.
\smallskip


REFERENCES ARE INDENTED\\
You've indented your references. Remove the \textbackslash begin\{quote\} and \textbackslash end{quote}
\smallskip


REFERENCES MALFORMED\\
You've used natbib, which is incompatible with aaai.bst.

As an alternate to this, try this workaround:

\begin{quote}\begin{scriptsize}\begin{verbatim}% \usepackage{natbib}
\newcommand{\citet}[1]
{\citeauthor{#1}~\shortcite{#1}}
\newcommand{\citep}{\cite}
\newcommand{\citealp}[1]
{\citeauthor{#1}~\citeyear{#1}}\end{verbatim}\end{scriptsize}\end{quote}
\smallskip


REFERENCES TOO SMALL\\
Your references are too small. They cannot be any smaller than 9 pt (\textbackslash small). Please correct this.
 \smallskip


SAVETREES\\
The savetrees may not be used.
\smallskip


SETLENGTH\\
You've used \textbackslash setlength to alter textfloats. This command is specifically disallowed. Please strip it from your source and recompile.
\smallskip


SETSPACE\\
The setspace package may not be used.
\smallskip


SPACE ABOVE, BELOW ELEMENTS ALTERED\\
You've adjusted the space above or below tables, figures, captions, floats, and/or section/subsection/subsubsections. These changes alter aaai.sty and are not allowed. Please remove them and either edit your text or resize figures or both.
\smallskip


SOURCE FILES MISSING\\
Your compressed archive does not contain *ALL* your source files (a stand-alone archive containing all the files necessary to recompile your paper in LaTeX). Consequently, your LaTeX file will not compile. You must create another compressed archive in which you have included all the files necessary to compile your paper on a separate computer. We cannot publish your paper without these files.
\smallskip


SPACE ALTERED ABOVE, BELOW FLOATS AND SECTIONS\\
You've adjusted the space above or below tables, figures, captions, and/or section/subsection/subsubsections. These changes alter aaai.sty and have resulted in a paper that is more difficult to read. Please remove them and either edit your text or resize figures or both.
\smallskip


STYLE FILE OF BIBLIOGRAPHY IS INCORRECT\\
You did not use aaai.bst (you used named.bst instead). This is not an IJCAI conference. You need to recompile your bibliography using the correct style for AAAI-18.
\smallskip


STYLE FILE IS INCORRECT\\
You did not use aaai18.sty. This style file is required. No other style file (including previous versions of aaai.sty) may be used.
\smallskip


STYLE FILE NOT IN CTAN\\
All styles must be available in the CTAN archive. If they are custom, you need to remove them and put the contents of the custom style directly in your preamble so that our source checker will allow compilation of your file.
\smallskip


TIMES PACKAGE NOT USED\\
Your paper is not formatted using the Times package, either because it is missing completely from your preamble, or because you are using another package that is cancelling use of times. This package is required.
\smallskip


TITLESEC\\
The title sec package may not be used.
\smallskip


VSPACE USED\\
You've used negative vspace around section / subsections. These must be removed.
\smallskip


TIMES PACKAGE MISSING\\
You neglected to use \textbackslash usepackage\{times\}. This font is required.
\smallskip


TITLE AND AUTHOR INFORMATION MALFORMED\\
You need to remove the formatting commands from your title and author information. The style for this information is governed by the conference style file. It cannot be altered.
\smallskip


TITLE BOX TOO SMALL\\
Your author-title-affiliation data exceeds the limit of the title box. To
fix this, you need to increase the size of the title box. You can do this by
placing
\textbackslash setlength\textbackslash titlebox\{2.5in\} in your preamble. You may need to increase the measurement until
the log file error goes away. (Please note: This command cannot be used elsewhere
in the paper, and cannot be used to shrink the size of the title box.)
\smallskip


TITLE IN SENTENCE CASE\\
Your title is formatted in sentence case. Title Case (Mixed Case) is required. Please capitalize the initial letters of all words in your title except for conjunctions and prepositions. You need to do this in your source, in the \textbackslash pdfinfo in your source, and on the submission website.
\smallskip


TITLE MALFORMED (TITLESEC)\\
You've used titlesec.sty, which alters the AAAI style. It must be removed.
\smallskip


TOCBIBIND\\
The tocbibind package may not be used.
\smallskip


ULEM\\
The ulem package may not be used.
\smallskip


T1ENC\\
The T1enc package is obsolete. Us the CM Super Fonts package instead.



\end{quote}
\end{small}


\section{Producing Reliable PDF\\Documents with \LaTeX{}}
Generally speaking, PDF files are platform independent and accessible to everyone. When creating a paper for a proceedings or publication in which many PDF documents must be merged and then printed on high-resolution PostScript RIPs, several requirements must be met that are not normally of concern. Thus to ensure that your paper will look like it does when printed on your own machine, you must take several precautions:
\begin{itemize}
\item Use type 1 fonts (not type 3 fonts)
\item Use only standard Times, Nimbus, and CMR font packages (not fonts like F3 or fonts with tildes in the names or fonts---other than Computer Modern---that are created for specific point sizes, like Times\~{}19) or fonts with strange combinations of numbers and letters
\item Embed all fonts when producing the PDF
\item Do not use the [T1]{fontenc} package (install the CM super fonts package instead)
\end{itemize}

\subsection{Creating Output Using PDF\LaTeX{} Is Required}
By using the PDF\TeX{} program instead of straight \LaTeX{} or \TeX{}, you will probably avoid the type 3 font problem altogether (unless you use a package that calls for metafont). PDF\LaTeX{} enables you to create a PDF document directly from \LaTeX{} source. The one requirement of this software is that all your graphics and images must be available in a format that PDF\LaTeX{} understands (normally PDF, jpg, or png).

PDF\LaTeX{}'s default is to create documents with type 1 fonts. If you find that it is not doing so in your case, it is likely that one or more fonts are missing from your system or are not in a path that is known to PDF\LaTeX{}.

\subsubsection{dvipdf Script}
Scripts such as dvipdf which ostensibly bypass the Postscript intermediary should not be used since they generally do not instruct dvips to use the config.pdf file.

\subsubsection{dvipdfm}
Do not use this dvi-PDF conversion package.

\subsection{Ghostscript}
\LaTeX{} users should not use GhostScript to create their PDFs.

\subsection{Graphics}
If you are still finding type 3 fonts in your PDF file, look at your graphics! \LaTeX{} users should check all their imported graphics files as well for font problems.

\section{Proofreading Your PDF}
Please check all the pages of your PDF file. Is the page size A4? Are there any type 3, Identity-H, or CID fonts? Are all the fonts embedded? Are there any areas where equations or figures run into the margins? Did you include all your figures? Did you follow mixed case capitalization rules for your title? Did you include a copyright notice? Do any of the pages scroll slowly (because the graphics draw slowly on the page)? Are URLs underlined and in color? You will need to fix these common errors before submitting your file. 

\section{Improperly Formatted Files }
In the past, AAAI has corrected improperly formatted files submitted by the authors. Unfortunately, this has become an increasingly burdensome expense that we can no longer absorb. Consequently, if your file is improperly formatted, it will not be included in the publication. If time allows, however, you will be notified via e-mail of the problems with your file and given the option of correcting the file yourself. A resubmission fee (minimum \$50.00, and likely higher) will be required for this service). Optionally, you may ask that AAAI have the file corrected for you, for an additional fee. If you opt to correct the file yourself, please note that we cannot provide you with any additional advice beyond that given in your packet. Files that are not corrected after a second attempt will not be included in the publication.

\subsection{\LaTeX{} 209 Warning}
If you use \LaTeX{} 209 we will not be able to publish your paper. Convert your paper to \LaTeX{}2e.

\section{Naming Your Electronic File}
We request that you name your \LaTeX{} source file with your last name (family name) so that it can easily be differentiated from other submissions. If you name your files with the name of the event or ``aaai" or ``paper" or ``camera-ready" or some other generic or indecipherable name, you bear all risks of loss --- it is extremely likely that your file may be overwritten.

\section{Submitting Your Electronic Files to AAAI}
Submitting your files to AAAI is a two-step process. It is explained fully in the author registration and submission instructions. Please consult this document for details on how to submit your paper.

\section{Inquiries} 
If you have any questions about the preparation or submission of your paper as instructed in this document, please contact AAAI Press at the address given below. If you have technical questions about implementation of the aaai style file, please contact an expert at your site. We do not provide technical support for \LaTeX{} or any other software package. To avoid problems, please keep your paper simple, and do not incorporate complicated macros and style files.

\begin{quote}
\noindent AAAI Press\\
2275 East Bayshore Road, Suite 160\\
Palo Alto, California 94303\\ 
\textit{Telephone:} (650) 328-3123\\ 
\textit{E-mail:} See the submission instructions for your particular conference or event.
\end{quote}

\section{Additional Resources}
\LaTeX{} is a difficult program to master. If you've used that software, and this document didn't help or some items were not explained clearly, we recommend you read Michael Shell's excellent document (testflow doc.txt V1.0a 2002/08/13) about obtaining correct PS/PDF output on \LaTeX{} systems. (It was written for another purpose, but it has general application as well). It is available at www.ctan.org in the tex-archive.

\section{ Acknowledgments}
AAAI is especially grateful to Peter Patel Schneider for his work in implementing the aaai.sty file, liberally using the ideas of other style hackers, including Barbara Beeton. We also acknowledge with thanks the work of George Ferguson for his guide to using the style and BibTeX files --- which has been incorporated into this document  --- and Hans Guesgen, who provided several timely modifications, as well as the many others who have, from time to time, sent in suggestions on improvements to the AAAI style. 

The preparation of the \LaTeX{} and Bib\TeX{} files that implement these instructions was supported by Schlumberger Palo Alto Research, AT\&T Bell Laboratories, Morgan Kaufmann Publishers, The Live Oak Press, LLC, and AAAI Press. Bibliography style changes were added by Sunil Issar. \verb+\+pubnote was added by J. Scott Penberthy. George Ferguson added support for printing the AAAI copyright slug. Additional changes to aaai.sty and aaai.bst have been made by the AAAI staff.

\bigskip
\noindent Thank you for reading these instructions carefully. We look forward to receiving your electronic files!

\end{document}

\def\year{2017}\relax
%File: formatting-instruction.tex
\documentclass[letterpaper]{article} %DO NOT CHANGE THIS
\usepackage{aaai17}  %Required
\usepackage{times}  %Required
\usepackage{helvet}  %Required
\usepackage{courier}  %Required
\usepackage{url}  %Required
\usepackage{graphicx}  %Required
\frenchspacing  %Required
\setlength{\pdfpagewidth}{8.5in}  %Required
\setlength{\pdfpageheight}{11in}  %Required
%PDF Info Is Required:
  \pdfinfo{
/Title (2017 Formatting Instructions for Authors Using LaTeX)
/Author (AAAI Press Staff)}
\setcounter{secnumdepth}{0}  
 \begin{document}
% The file aaai.sty is the style file for AAAI Press 
% proceedings, working notes, and technical reports.
%
\title{2017 Formatting Instructions \\for Authors Using \LaTeX{}}
\author{AAAI Press\\
Association for the Advancement of Artificial Intelligence\\
2275 East Bayshore Road, Suite 160\\
Palo Alto, California 94303\\
}
\maketitle
\begin{abstract}
AAAI creates proceedings, working notes, and technical reports directly from electronic source furnished by the authors. To ensure that all papers in the publication have a uniform appearance, authors must adhere to the following instructions. 
\end{abstract}

\noindent Congratulations on having a paper selected for inclusion in an AAAI Press proceedings or technical report! This document details the requirements necessary to get your accepted paper published using \LaTeX{}. If you are using Microsoft Word, instructions are provided in a different document. If you want to use some other formatting software, you must obtain permission from AAAI Press first. 

The instructions herein are provided as a general guide for experienced \LaTeX{} users. If you do not know how to use \LaTeX{}, do not use it to format your paper. AAAI cannot provide you with support and the accompanying style files are \textbf{not} guaranteed to work. If the results you obtain are not in accordance with the specifications you received, you must correct your source file to achieve the correct result. 

These instructions are generic. Consequently, they do not include specific dates, page charges, and so forth. Please consult your specific written conference instructions for details regarding your submission. Please review the entire document for specific instructions that might apply to your particular situation. All authors must comply with the following:

\begin{itemize}
\item You must use the latest AAAI Press \LaTeX{} style file and bib file, which are located in the current author kit.
\item You must complete, sign, and return by the deadline the AAAI copyright form (proceedings authors) or distribution license (technical report authors).
\item You must read and format your paper source and PDF according to the formatting instructions for authors.
\item You must submit your electronic files and abstract using our electronic submission form \textbf{on time.}
\item You must pay any required page or formatting charges to AAAI Press so that they are received by the deadline.
\item You must check your paper before submitting it, ensuring that it compiles without error, and complies with the guidelines found in the author kit.
\end{itemize}

\section{Copyright}
All papers submitted for publication by AAAI Press must be accompanied by a valid signed copyright form or, in the case of technical reports, by a valid signed permission to distribute form. There are no exceptions to this requirement. You must send us the original version of this form. However, to meet the deadline, you may fax (1-650-321-4457) or scan and e-mail the form (pubforms17@aaai.org) to AAAI by the submission deadline, and then mail the original via postal mail to the AAAI office. \textbf{If you fail to send in a signed copyright or permission form, we will be unable to publish your paper. There are no exceptions to this policy.} You will find PDF versions of the AAAI copyright and permission to distribute forms in the author kit.

\section{Formatting Requirements in Brief}
We need source and PDF files that can be used in a variety of ways and can be output on a variety of devices. The design and appearance of the paper is governed by the aaai style file. You must not make any changes to this file, nor use any commands, packages, style files, or macros that alter that design, including, but not limited to spacing, floats, margins, fonts, font size, and appearance. AAAI imposes  requirements on your source and PDF files that must be followed. Most of these requirements are based on our efforts to standardize conference manuscript properties and layout. All papers submitted to AAAI for publication must comply with the following:

\begin{itemize}
\item Your .tex file must compile in PDF\LaTeX{} --- \textbf{ no .ps or .eps figure files.}
\item All fonts must be embedded in the PDF file --- \textbf{ this includes your figures.}
\item Modifications to the style file, whether directly or via commands in your document may not be made, most especially when made in an effor to avoid extra page charges or make your paper fit in a specific number of pages.
\item No type 3 fonts may be used (even in illustrations).
\item You may not alter the spacing above and below captions, figures, headings, and subheadings.
\item You may not alter the font sizes of text elements, footnotes, heading elements, captions, or title information (for references and tables and mathematics, please see the the limited exceptions provided herein).
\item You may not alter the line spacing of text.
\item Your title must follow Title Case capitalization rules (not sentence case).
\item Your .tex file include completed metadata to pass-through to the PDF (see PDFINFO below)
\item \LaTeX{} documents must use the Times or Nimbus font package (do not use Computer Modern for the text of your paper).
\item No \LaTeX{} 209 documents may be used or submitted.
\item Your source must not require use of fonts for non-Roman alphabets within the text itself. If your paper includes symbols in other languages (such as, but not limited to Arabic, Chinese, Hebrew, Japanese, Russian and other Cyrillic languages), you must restrict their use to figures.
\item Fonts that require non-English language support (CID and Identity-H) must be converted to outlines or 300 dpi bitmap or removed from the document (even if they are in a graphics file embedded in the document). 
\item Two-column format in AAAI style is required for all papers.
\item The paper size for final submission must be US letter without exception.
\item The source file must exactly match the PDF.
\item The document margins must be as specified in the formatting instructions.
\item The number of pages and the file size must be as specified for your event.
\item No document may be password protected.
\item Neither the PDFs nor the source may contain any embedded links or bookmarks.
\item Your source and PDF must not have any page numbers, footers, or headers.
\item Your PDF must be compatible with Acrobat 5 or higher.
\item Your \LaTeX{} source file (excluding references) must consist of a \textbf{single} file (use of the ``input" command is not allowed.
\item Your graphics must be sized appropriately outside of \LaTeX{} (do not use the ``clip" command) .
\end{itemize}

If you do not follow the above requirements, it is likely that we will be unable to publish your paper.

\section{What Files to Submit}
You must submit the following items to ensure that your paper is published:
\begin{itemize}
\item A fully-compliant PDF file.
\item Your  \LaTeX{}  source file submitted as a \textbf{single} .tex file (do not use the ``input" command to include sections of your paper --- every section must be in the single source file). The only exception is the reference list, which you should include separately. Your source must compile on our system, which includes the standard \LaTeX{} support files.
\item Only the graphics files used in compiling paper.
\item The \LaTeX{}-generated files (e.g. .aux and .bib file, etc.) for your compiled source.
\item If you have used an old installation of \LaTeX{}, you should include algorithm style files). If in doubt, include it.
\end{itemize}

Your \LaTeX{} source will be reviewed and recompiled on our system (if it does not compile, you may incur late fees).   \textbf{Do not submit your source in multiple text files.} Your single \LaTeX{} source file must include all your text, your bibliography (formatted using aaai.bst), and any custom macros. Accompanying this source file, you must also supply any nonstandard (or older) referenced style files and all your referenced graphics files. 

Your files should work without any supporting files (other than the program itself) on any computer with a standard \LaTeX{} distribution. Place your PDF and source files in a single tar, zipped, gzipped, stuffed, or compressed archive. Name your source file with your last (family) name.

\textbf{Do not send files that are not actually used in the paper.} We don't want you to send us any files not needed for compiling your paper, including, for example, this instructions file, unused graphics files, standard style files, and so forth.

\textbf{Obsolete style files.}  The commands for some common packages (such as some used for algorithms), may have changed. Please be certain that you are not compiling your paper using old or obsolete style files. 

\section{Using \LaTeX{} to Format Your Paper}

The latest version of the AAAI style file is available on AAAI's website. Download this file and place it in  the \TeX\ search path. Placing it in the same directory as the paper should also work. You must download the latest version of the complete author kit so that you will have the latest instruction set and style file.

\subsection{Document Preamble}

In the \LaTeX{} source for your paper, you \textbf{must} place the following lines as shown in the example in this subsection. This command set-up is for three authors. Add or subtract author and address lines as necessary, and uncomment the portions that apply to you. In most instances, this is all you need to do to format your paper in the Times font. The helvet package will cause Helvetica to be used for sans serif. These files are part of the PSNFSS2e package, which is freely available from many Internet sites (and is often part of a standard installation).

Leave the setcounter for section number depth commented out and set at 0 unless you want to add section numbers to your paper. If you do add section numbers, you must uncomment this line and change the number to 1 (for section numbers), or 2 (for section and subsection numbers). The style file will not work properly with numbering of subsubsections, so do not use a number higher than 2.

If (and only if) your author title information will not fit within the specified  height allowed, put \textbackslash setlength \textbackslash titlebox{2.5in} in your preamble. Increase the height until the height error disappears from your log. You may not use the  \textbackslash setlength command elsewhere in your paper, and it may not be used to reduce the height of the author-title box.

\subsubsection{The Following Must Appear in Your Preamble}
\begin{quote}
\begin{scriptsize}\begin{verbatim}
\documentclass[letterpaper]{article}
\usepackage{aaai}
\usepackage{times}
\usepackage{helvet}
\usepackage{courier}
\usepackage{url}
\usepackage{graphicx}
\frenchspacing
% Add additional packages here. The following
% packages may NOT be used  (this list
% is not exhaustive:
% authblk, caption, CJK, float, fullpage, geometry, 
%hyperref, layout, nameref, natbib, savetrees, 
%setspace, titlesec, tocbibind, ulem
%
%US Lettersize Paper Is Required
\setlength{pdfpagewidth}{8.5in}
\setlength{pdfpageheight}{11in}\\
%
%
% PDFINFO
% You are required to complete the following
% for pass-through to the PDF. 
% No LaTeX commands of any kind may be
% entered. The parentheses and spaces 
% are an integral part of the 
% pdfinfo script and must not be removed.
%
\pdfinfo{
/Title (Input Your Paper Title Here)
/Author (John Doe, Jane Doe)
/Keywords (Input your keywords in this optional area)
}
%
%Section Numbers
% Uncomment if you want to use section numbers
% and change the 0 to a 1 or 2
% \setcounter{secnumdepth}{0}

% Title and Author Information Must Immediate Follow
% the pdfinfo within the preamble
%
\title{Title}\\
\author\{Author 1 \ and Author 2\\
Address line\\
Address line\\
\ And\\
Author 3\\
Address line\\
Address line
}\\
%
\end{verbatim}\end{scriptsize}
\end{quote}

\subsection{Preparing Your Paper}

After the preamble above, you should prepare your paper as follows:
\begin{quote}
\begin{scriptsize}\begin{verbatim}
%
\begin{document}
\maketitle
\begin{abstract}
%...
\end{abstract}\end{verbatim}\end{scriptsize}
\end{quote}

\subsubsection{The Following Must Conclude Your Document}
\begin{quote}
\begin{scriptsize}\begin{verbatim}
%References and End of Paper
%These lines must be placed at the end of your paper
\bibliography{Bibliography-File}
\bibliographystyle{aaai}
\end{document}
\end{verbatim}\end{scriptsize}
\end{quote}

\subsection{Inserting Document Metadata with \LaTeX{}}
PDF files contain document summary information that enables us to create an Acrobat index (pdx) file, and also allows search engines to locate and present your paper more accurately. \textit{Document metadata  for author and title are REQUIRED.} 

\textit{Important:} Do not include \textit{any} \LaTeX{} code or nonascii characters (including accented characters) in the metadata. The data in the metadata must be completely plain ascii. It may not include slashes, accents, linebreaks, unicode, or any \LaTeX{} commands. Type the title exactly as it appears on the paper (minus all formatting). Input the author names in the order in which they appear on the paper (minus all accents), separating each author by a comma. You may also include keywords in the Keywords field.




\begin{quote}
\begin{small}
\textbackslash begin\{document\}\\
\textbackslash maketitle\\
...\\
\textbackslash bibliography\{Bibliography-File\}\\
\textbackslash bibliographystyle\{aaai\}\\
\textbackslash end\{document\}\\
\end{small}
\end{quote}
\subsection{Incompatible Packages}
The following packages are incompatible with aaai.sty and/or aaai.bst and must not be used (this list is not exhaustive --- there are others as well):
\begin{itemize}
\item authblk
\item caption 
\item CJK
\item float
\item fullpage
\item geometry
\item hyperref
\item layout
\item nameref
\item natbib
\item savetrees
\item setspace
\item titlesec
\item tocbibind
\item ulem
\item T1 fontenc package (install the CM super fonts package instead)
\end{itemize}


\subsection{Illegal Commands}
The following commands may not be used in your paper (this list is not exhaustive --- there are others; generally, if it alters floats, margins, fonts, linespacing, or the presentation of the references and citations, it is unacceptable:
\begin{itemize}
\item \textbackslash renewcommand (in almost all instances)
\item baselinestretch
\item \textbackslash setlength (except for titlebox)
\item \textbackslash input
\item \textbackslash vspace or vskip (when used before or after a section or subsection)
\item \textbackslash addtolength 
\item \textbackslash columnsep
\item \textbackslash top margin (or text height or addsidemargin or even side margin)
\item trim or clip (used to crop figures)
\item any command that globally alters floats, space above and below figures and tables
\end{itemize}

\subsection{Illegal Commands in Final Paper}
For your final camera ready copy, you must not use any page break commands, including, but not limited to:
\begin{itemize}
\item \textbackslash newpage
\item \textbackslash break
\item \textbackslash clearpage
\item \textbackslash pagebreak
\end{itemize}
(References must flow directly after the text without breaks.) Note that this may \textit{not} the case when submitting a paper for review. Some conferences require references to be on a separate page during the review process. AAAI Press, however, does not require this condition for the final paper.


\subsection{Paper Size, Margins, and Column Width}
Papers must be formatted to print in two-column format on 8.5 x 11 inch US letter-sized paper. The margins must be exactly as follows: 
\begin{itemize}
\item Top margin: .75 inches
\item Left margin: .75 inches
\item Right margin: .75 inches
\item Bottom margin: 1.25 inches
\end{itemize} 


The default paper size in most installations of \LaTeX{} is A4. However, because we require that your electronic paper be formatted in US letter size, you will need to alter the default for this paper to US letter size. Assuming you are using the 2e version of \LaTeX{}, you can do this by including the [letterpaper] option at the beginning of your file: 
\textbackslash documentclass[letterpaper]{article}. 

This command is usually sufficient to change the format. Sometimes, however, it may not work. Use PDF\LaTeX{} and include
\textbackslash setlength\{\textbackslash pdfpagewidth\}\{8.5in\}
\textbackslash setlength\{\textbackslash pdfpageheight\}\{11in\}
in your preamble. 

\textbf{Do not use the Geometry package to alter the page size.} Use of this style file alters aaai.sty and will result in your paper being rejected. 


\subsubsection{Column Width and Margins.}
To ensure maximum readability, your paper must include two columns. Each column should be 3.3 inches wide (slightly more than 3.25 inches), with a .375 inch (.952 cm) gutter of white space between the two columns. The aaai.sty file will automatically create these columns for you. 

\subsection{Overlength Papers}
If your paper is too long, turn on \textbackslash frenchspacing, which will reduce the space after periods. Next,  shrink the size of your graphics. Use \textbackslash centering instead of \textbackslash begin\{center\} in your figure environment. For mathematical environments, you may reduce fontsize. You may also alter the size of your bibliography by inserting \textbackslash fontsize\{9.5pt\}\{10.5pt\} \textbackslash selectfont
right before the bibliography (the minimum size is \textbackslash fontsize\{9.0pt\}\{10.0pt\}. 

Commands that alter page layout are forbidden. These include \textbackslash columnsep, \textbackslash topmargin, \textbackslash topskip, \textbackslash textheight, \textbackslash textwidth, \textbackslash oddsidemargin, and \textbackslash evensizemargin (this list is not exhaustive). If you alter page layout, you will be required to pay the page fee \textit{plus} a reformatting fee. Other commands that are questionable and may cause your paper to be rejected include  \textbackslash parindent, and \textbackslash parskip. Commands that alter the space between sections are forbidden. The title sec package is not allowed. Regardless of the above, if your paper is obviously ``squeezed" it is not going to to be accepted. Before using every trick you know to make your paper a certain length, try reducing the size of your graphics or cutting text instead or (if allowed) paying the extra page charge.

\subsection{Figures}
Your paper must compile in PDF\LaTeX{}. Consequently, all your figures must be .jpg, .png, or .pdf. You may not use  the .gif (the resolution is too low), .ps, or .eps file format for your figures.

When you include your figures, you must crop them \textbf{outside} of \LaTeX{}. The command \textbackslash includegraphics*[clip=true, viewport 0 0 10 10]{...} might result in a PDF that looks great, but the image is \textbf{not really cropped.} The full image can reappear (and obscure whatever it is overlapping) when page numbers are applied or color space is standardized. 

\subsection{Type Font and Size}
Your paper must be formatted in Times Roman or Nimbus. We will not accept papers formatted using Computer Modern or Palatino or some other font as the text or heading typeface. Sans serif, when used, should be Courier. Use Symbol or Lucida or Computer Modern for \textit{mathematics only. } 

Do not use type 3 fonts for any portion of your paper, including graphics. Type 3 bitmapped fonts are designed for fixed resolution printers. Most print at 300 dpi even if the printer resolution is 1200 dpi or higher. They also often cause high resolution imagesetter devices and our PDF indexing software to crash. Consequently, AAAI will not accept electronic files containing obsolete type 3 fonts. Files containing those fonts (even in graphics) will be rejected. 

Fortunately, there are effective workarounds that will prevent your file from embedding type 3 bitmapped fonts. The easiest workaround is to use the required times, helvet, and courier packages with \LaTeX{}2e. (Note that papers formatted in this way will still use Computer Modern for the mathematics. To make the math look good, you'll either have to use Symbol or Lucida, or you will need to install type 1 Computer Modern fonts --- for more on these fonts, see the section ``Obtaining Type 1 Computer Modern.")

If you are unsure if your paper contains type 3 fonts, view the PDF in Acrobat Reader. The Properties/Fonts window will display the font name, font type, and encoding properties of all the fonts in the document. If you are unsure if your graphics contain type 3 fonts (and they are PostScript or encapsulated PostScript documents), create PDF versions of them, and consult the properties window in Acrobat Reader. 

The default size for your type should be ten-point with twelve-point leading (line spacing). Start all pages (except the first) directly under the top margin. (See the next section for instructions on formatting the title page.) Indent ten points when beginning a new paragraph, unless the paragraph begins directly below a heading or subheading.

\subsubsection{Obtaining Type 1 Computer Modern for \LaTeX{}.}

If you use Computer Modern for the mathematics in your paper (you cannot use it for the text) you may need to download type 1 Computer fonts. They are available without charge from the American Mathematical Society:
http://www.ams.org/tex/type1-fonts.html. 

\subsection{Title and Authors}
Your title must appear in mixed case (nouns, pronouns, and verbs are capitalized) near the top of the first page, centered over both columns in sixteen-point bold type (twenty-four point leading). This style is called ``mixed case." Author's names should appear below the title of the paper, centered in twelve-point type (with fifteen point leading), along with affiliation(s) and complete address(es) (including electronic mail address if available) in nine-point roman type (the twelve point leading). (If the title is long, or you have many authors, you may reduce the specified point sizes by up to two points.) You should begin the two-column format when you come to the abstract. 

\subsubsection{Formatting Author Information}
Author information can be set in a number of different styles, depending on the number of authors and the number of affiliations you need to display. For several authors from the same institution, use \textbackslash and:

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{Author 1 \and ... \and Author n\\ 
Address line \\ ... \\ Address line}
\end{verbatim}\end{scriptsize}\end{quote}

\noindent If the names do not fit well on one line use:

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{Author 1} ... \\
{\bf \Large Author  ...  Author}\\ 
Address line \\ ... \\ Address line
}
\end{verbatim}\end{scriptsize}\end{quote}

\noindent For authors from different institutions, use \textbackslash And:

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{Author 1\\ Address line \\ ... \\ Address line 
\And ... \And Author n\\
Address line\\ ... \\ Address line}
\end{verbatim}\end{scriptsize}\end{quote}

\noindent To start a separate ``row" of authors, use \textbackslash AND:

\noindent If the title and author information does not fit in the area allocated, place
\textbackslash setlength\textbackslash titlebox\{\textit{height}\}
after the \textbackslash documentclass line where \{\textit{height}\} is  2.5in or greater.

\subsubsection{Formatting Author Information --- Alternative Method}
If your paper has a large number of authors from different institutions, you may use the following alternative method for displaying the author information.

\begin{quote}\begin{scriptsize}\begin{verbatim}
\author{AuthorOne},\textsuperscript{1}
\author{AuthorTwo},\textsuperscript{2}
\author{AuthorThree},\textsuperscript{3}
\author{AuthorFour},\textsuperscript{4}
\author{AuthorFive}, \textsuperscript{5}\\
\textsuperscript{1}AffiliationOne}\\
\textsuperscript{2}AffiliationTwo}\\
\textsuperscript{3}AffiliationThree}\\
\textsuperscript{4}AffiliationFour}\\
\textsuperscript{5}AffiliationFive}\\
\{email, email\}@affiliation.com,
email@affiliation.com,
email@affiliation.com, 
email@affiliation.com
\end{verbatim}\end{scriptsize}\end{quote}


\subsection{\LaTeX{} Copyright Notice}
The copyright notice automatically appears if you use aaai.sty. If you are creating a technical report, it is not necessary to include this notice. You may disable the copyright line using the \verb+\+nocopyrightcommand. To change the entire text of the copyright slug, use:
\textbackslash copyrighttext \{\emph{text}\}.
Either of these must appear before \textbackslash maketitle. Please be advised, however, that \textit{if you disable or change the copyright line and transfer of copyright is required, your paper will not be published.}

\subsection{Credits}
Any credits to a sponsoring agency should appear in the acknowledgments section, unless the agency requires different placement. If it is necessary to include this information on the front page, use
\textbackslash thanks in either the \textbackslash author or \textbackslash title commands.
For example:
\begin{quote}
\begin{small}
\textbackslash title\{Very Important Results in AI\textbackslash thanks\{This work is
 supported by everybody.\}\}
\end{small}
\end{quote}
Multiple \textbackslash thanks commands can be given. Each will result in a separate footnote indication in the author or title with the corresponding text at the botton of the first column of the document. Note that the \textbackslash thanks command is fragile. You will need to use \textbackslash protect.

Please do not include \textbackslash pubnote commands in your document.

\subsection{Abstract}
Follow the example commands in this document for creation of your abstract. Further indentation is not required. {Do not include references in your abstract!}

\subsection{Page Numbers}

Do not \textbf{ever} print any page numbers on your paper. 

\subsection{Text }
The main body of the paper must be formatted in ten-point with twelve-point leading (line spacing).

\subsection{Citations}
Citations within the text should include the author's last name and year, for example (Newell 1980). Append lower-case letters to the year in cases of ambiguity. Multiple authors should be treated as follows: (Feigenbaum and Engelmore 1988) or (Ford, Hayes, and Glymour 1992). In the case of four or more authors, list only the first author, followed by et al. (Ford et al. 1997).

\subsection{Extracts}
Long quotations and extracts should be indented ten points from the left and right margins. 

\begin{quote}
This is an example of an extract or quotation. Note the indent on both sides. Quotation marks are not necessary if you offset the text in a block like this, and properly identify and cite the quotation in the text. 

\end{quote}

\subsection{Footnotes}
Avoid footnotes as much as possible; they interrupt the reading of the text. When essential, they should be consecutively numbered throughout with superscript Arabic numbers. Footnotes should appear at the bottom of the page, separated from the text by a blank line space and a thin, half-point rule. 

\subsection{Headings and Sections}
When necessary, headings should be used to separate major sections of your paper. Remember, you are writing a short paper, not a lengthy book! An overabundance of headings will tend to make your paper look more like an outline than a paper. The aaai.sty package will create headings for you. Do not alter their size nor their spacing above or below. 

\subsubsection{Section Numbers}
The use of section numbers in AAAI Press papers is optional. To use section numbers in \LaTeX{}, uncomment the setcounter line in your document preamble and change the 0 to a 1 or 2. Section numbers should not be used in short poster papers.

\subsubsection{Section Headings.}
Sections should be arranged and headed as follows: 

\subsubsection{Acknowledgments.}
The acknowledgments section, if included, appears after the main body of text and is headed ``Acknowledgments." This section includes acknowledgments of help from associates and colleagues, credits to sponsoring agencies, financial support, and permission to publish. Please acknowledge other contributors, grant support, and so forth, in this section. Do not put acknowledgments in a footnote on the first page. If your grant agency requires acknowledgment of the grant on page 1, limit the footnote to the required statement, and put the remaining acknowledgments at the back. Please try to limit acknowledgments to no more than three sentences. 

\subsubsection{Appendices.}
Any appendices follow the acknowledgments, if included, or after the main body of text if no acknowledgments appear. 

\subsubsection{References}
The references section should be labeled ``References" and should appear at the very end of the paper (don't end the paper with references, and then put a figure by itself on the last page). A sample list of references is given later on in these instructions. Please use a consistent format for references. Poorly prepared or sloppy references reflect badly on the quality of your paper and your research. Please prepare complete and accurate citations.

\subsection{Illustrations and Figures}
Figures, drawings, tables, and photographs should be placed throughout the paper near the place where they are first discussed. Do not group them together at the end of the paper. If placed at the top or bottom of the paper, illustrations may run across both columns. Figures must not invade the top, bottom, or side margin areas. Figures must be inserted using the \textbackslash usepackage\{graphicx\}. Number figures sequentially, for example, figure 1, and so on. 

The illustration number and caption should appear under the illustration. Labels, and other text with the actual illustration must be at least nine-point type. 

If your paper includes illustrations that are not compatible with PDF\TeX{} (such as .eps or .ps documents), you will need to convert them. The epstopdf package will usually work for eps files. You will need to convert your ps files to PDF however.



\subsubsection{Low-Resolution Bitmaps.}
You may not use low-resolution (such as 72 dpi) screen-dumps and GIF files---these files contain so few pixels that they are always blurry, and illegible when printed. If they are color, they will become an indecipherable mess when converted to black and white. This is always the case with gif files, which should never be used. The resolution of screen dumps can be increased by reducing the print size of the original file while retaining the same number of pixels. You can also enlarge files by manipulating them in software such as PhotoShop. Your figures should be 300 dpi when incorporated into your document.

\subsubsection{\LaTeX{} Overflow.}
\LaTeX{} users please beware: \LaTeX{} will sometimes put portions of the figure or table or an equation in the margin. If this happens, you need to scale the figure or table down, or reformat the equation.{ \bf Check your log file!} You must fix any overflow into the margin (that means no overfull boxes in \LaTeX{}). If you don't, the overflow text will simply be eliminated. \textbf{Nothing is permitted to intrude into the margin or gutter.}

\subsubsection{Using Color.}
Your paper will be printed in black and white and grayscale. Consequently, because conversion to grayscale can cause undesirable effects (red changes to black, yellow can disappear, and so forth), we strongly suggest you avoid placing color figures in your document. Of course, any reference to color will be indecipherable to your reader. 

\subsubsection{Drawings.}
We suggest you use computer drawing software (such as Adobe Illustrator or, (if unavoidable), the drawing tools in Microsoft Word) to create your illustrations. Do not use Microsoft Publisher. These illustrations will look best if all line widths are uniform (half- to two-point in size), and you do not create labels over shaded areas. Shading should be 133 lines per inch if possible. Use Times Roman or Helvetica for all figure call-outs. \textbf{Do not use hairline width lines} --- be sure that the stroke width of all lines is at least .5 pt. Zero point lines will print on a laser printer, but will completely disappear on the high-resolution devices used by our printers.

\subsubsection{Photographs and Images.}
Photographs and other images should be in grayscale (color photographs will not reproduce well; for example, red tones will reproduce as black, yellow may turn to white, and so forth) and set to a minimum of 300 dpi. Do not prescreen images.

\subsubsection{Resizing Graphics.}
Resize your graphics \textbf{before} you include them with LaTeX. You may \textbf{not} use trim or clip options as part of your \textbackslash includgraphics command. Resize the media box of your PDF using a graphics program instead. 

\subsubsection{Fonts in Your Illustrations}
You must embed all fonts in your graphics before including them in your LaTeX document.

\subsection{References} 
The AAAI style includes a set of definitions for use in formatting references with BibTeX. These definitions make the bibliography style fairly close to the one specified below. To use these definitions, you also need the BibTeX style file ``aaai.bst," available in the author kit on the AAAI web site. Then, at the end of your paper but before \textbackslash end{document}, you need to put the following lines:

\begin{quote}
\begin{small}
\textbackslash bibliographystyle\{aaai\}
\textbackslash bibliography\{bibfile1,bibfile2,...\}
\end{small}
\end{quote}

Please note that you are required to use \textbackslash bibliographystyle\{aaai\} for your references. You may not use named, plain, apalike, acm, ieeetr, siam, chicago, or any other style. Use of natbib is also not acceptable.

The list of files in the \textbackslash  bibliography command should be the names of your BibTeX source files (that is, the .bib files referenced in your paper).

The following commands are available for your use in citing references:
\begin{quote}
\begin{small}
\textbackslash cite: Cites the given reference(s) with a full citation. This appears as ``(Author Year)'' for one reference, or ``(Author Year; Author Year)'' for multiple references.\\
\textbackslash shortcite: Cites the given reference(s) with just the year. This appears as ``(Year)'' for one reference, or ``(Year; Year)'' for multiple references.\\
\textbackslash citeauthor: Cites the given reference(s) with just the author name(s) and no parentheses.\\
\textbackslash citeyear: Cites the given reference(s) with just the date(s) and no parentheses.
\end{small}
\end{quote}

\textbf{Warning:} The aaai.sty file is incompatible with the hyperref and natbib packages. If you use either, your references will be garbled and your paper will not be published.

Formatted bibliographies should look like the following examples.

\smallskip \noindent \textit{Book with Multiple Authors}\\
Engelmore, R., and Morgan, A. eds. 1986. \textit{Blackboard Systems.} Reading, Mass.: Addison-Wesley.

\smallskip \noindent \textit{Journal Article}\\
Robinson, A. L. 1980a. New Ways to Make Microcircuits Smaller. \textit{Science} 208: 1019--1026.

\smallskip \noindent \textit{Magazine Article}\\
Hasling, D. W.; Clancey, W. J.; and Rennels, G. R. 1983. Strategic Explanations in Consultation. \textit{The International Journal of Man-Machine Studies} 20(1): 3--19.

\smallskip \noindent \textit{Proceedings Paper Published by a Society}\\
Clancey, W. J. 1983b. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. In Proceedings of the Eighth International Joint Conference on Artificial Intelligence, 556--560. Menlo Park, Calif.: International Joint Conferences on Artificial Intelligence, Inc.

\smallskip \noindent \textit{Proceedings Paper Published by a Press or Publisher}\\
Clancey, W. J. 1984. Classification Problem Solving. In \textit{Proceedings of the Fourth National Conference on Artificial Intelligence,} 49--54. Menlo Park, Calif.: AAAI Press. 

\smallskip \noindent \textit{University Technical Report}\\
Rice, J. 1986. Poligon: A System for Parallel Problem Solving, Technical Report, KSL-86-19, Dept. of Computer Science, Stanford Univ. 

\smallskip \noindent \textit{Dissertation or Thesis}\\
Clancey, W. J. 1979b. Transfer of Rule-Based Expertise through a Tutorial Dialogue. Ph.D. diss., Dept. of Computer Science, Stanford Univ., Stanford, Calif.

\smallskip \noindent \textit{Forthcoming Publication}\\
Clancey, W. J. 1986a. The Engineering of Qualitative Models. Forthcoming.



\section{Producing Reliable PDF\\Documents with \LaTeX{}}
Generally speaking, PDF files are platform independent and accessible to everyone. When creating a paper for a proceedings or publication in which many PDF documents must be merged and then printed on high-resolution PostScript RIPs, several requirements must be met that are not normally of concern. Thus to ensure that your paper will look like it does when printed on your own machine, you must take several precautions:
\begin{itemize}
\item Use type 1 fonts (not type 3 fonts)
\item Use only standard Times, Nimbus, and CMR font packages (not fonts like F3 or fonts with tildes in the names or fonts---other than Computer Modern---that are created for specific point sizes, like Times\~{}19) or fonts with strange combinations of numbers and letters
\item Embed all fonts when producing the PDF
\item Do not use the [T1]{fontenc} package (install the CM super fonts package instead)
\end{itemize}

\subsection{Creating Output Using PDF\LaTeX{} Is Required}
By using the PDF\TeX{} program instead of straight \LaTeX{} or \TeX{}, you will probably avoid the type 3 font problem altogether (unless you use a package that calls for metafont). PDF\LaTeX{} enables you to create a PDF document directly from \LaTeX{} source. The one requirement of this software is that all your graphics and images must be available in a format that PDF\LaTeX{} understands (normally PDF, jpg, or png).

PDF\LaTeX{}'s default is to create documents with type 1 fonts. If you find that it is not doing so in your case, it is likely that one or more fonts are missing from your system or are not in a path that is known to PDF\LaTeX{}.

\subsubsection{dvipdf Script}
Scripts such as dvipdf which ostensibly bypass the Postscript intermediary should not be used since they generally do not instruct dvips to use the config.pdf file.

\subsubsection{dvipdfm}
Do not use this dvi-PDF conversion package.

\subsection{Ghostscript}
\LaTeX{} users should not use GhostScript to create their PDFs.

\subsection{Graphics}
If you are still finding type 3 fonts in your PDF file, look at your graphics! \LaTeX{} users should check all their imported graphics files as well for font problems.

\section{Proofreading Your PDF}
Please check all the pages of your PDF file. Is the page size A4? Are there any type 3, Identity-H, or CID fonts? Are all the fonts embedded? Are there any areas where equations or figures run into the margins? Did you include all your figures? Did you follow mixed case capitalization rules for your title? Did you include a copyright notice? Do any of the pages scroll slowly (because the graphics draw slowly on the page)? Are URLs underlined and in color? You will need to fix these common errors before submitting your file. 

\section{Improperly Formatted Files }
In the past, AAAI has corrected improperly formatted files submitted by the authors. Unfortunately, this has become an increasingly burdensome expense that we can no longer absorb. Consequently, if your file is improperly formatted, it will not be included in the publication. If time allows, however, you will be notified via e-mail (with a copy to the program chair) of the problems with your file and given the option of correcting the file yourself (and paying a late fee) or asking that AAAI have the file corrected for you, for an additional fee. If you opt to correct the file yourself, please note that we cannot provide you with any additional advice beyond that given in your packet. Files that are not corrected after a second attempt will not be included in the publication.

\subsection{\LaTeX{} 209 Warning}
If you use \LaTeX{} 209 we will not be able to publish your paper. Convert your paper to \LaTeX{}2e.

\section{Naming Your Electronic File}
We request that you name your \LaTeX{} source file with your last name (family name) so that it can easily be differentiated from other submissions. If you name your files with the name of the event or ``aaai" or ``paper" or ``camera-ready" or some other generic or indecipherable name, you bear all risks of loss --- it is extremely likely that your file may be overwritten.

\section{Submitting Your Electronic Files to AAAI}
Submitting your files to AAAI is a two-step process. It is explained fully in the author registration and submission instructions. Please consult this document for details on how to submit your paper.

\section{Inquiries} 
If you have any questions about the preparation or submission of your paper as instructed in this document, please contact AAAI Press at the address given below. If you have technical questions about implementation of the aaai style file, please contact an expert at your site. We do not provide technical support for \LaTeX{} or any other software package. To avoid problems, please keep your paper simple, and do not incorporate complicated macros and style files.

\begin{quote}
\noindent AAAI Press\\
2275 East Bayshore Road, Suite 160\\
Palo Alto, California 94303\\ 
\textit{Telephone:} (650) 328-3123\\ 
\textit{E-mail:} See the submission instructions for your particular conference or event.
\end{quote}

\section{Additional Resources}
\LaTeX{} is a difficult program to master. If you've used that software, and this document didn't help or some items were not explained clearly, we recommend you read Michael Shell's excellent document (testflow doc.txt V1.0a 2002/08/13) about obtaining correct PS/PDF output on \LaTeX{} systems. (It was written for another purpose, but it has general application as well). It is available at www.ctan.org in the tex-archive.

\section{ Acknowledgments}
AAAI is especially grateful to Peter Patel Schneider for his work in implementing the aaai.sty file, liberally using the ideas of other style hackers, including Barbara Beeton. We also acknowledge with thanks the work of George Ferguson for his guide to using the style and BibTeX files --- which has been incorporated into this document  --- and Hans Guesgen, who provided several timely modifications, as well as the many others who have, from time to time, sent in suggestions on improvements to the AAAI style. 

The preparation of the \LaTeX{} and Bib\TeX{} files that implement these instructions was supported by Schlumberger Palo Alto Research, AT\&T Bell Laboratories, Morgan Kaufmann Publishers, The Live Oak Press, LLC, and AAAI Press. Bibliography style changes were added by Sunil Issar. \verb+\+pubnote was added by J. Scott Penberthy. George Ferguson added support for printing the AAAI copyright slug. Additional changes to aaai.sty and aaai.bst have been made by the AAAI staff.

\bigskip
\noindent Thank you for reading these instructions carefully. We look forward to receiving your electronic files!

\end{document}


\documentclass[sigconf]{acmart}

\usepackage{booktabs} % For formal tables

\usepackage{algorithm}
\usepackage{algpseudocode}


% Copyright
%\setcopyright{none}
%\setcopyright{acmcopyright}
%\setcopyright{acmlicensed}
\setcopyright{rightsretained}
%\setcopyright{usgov}
%\setcopyright{usgovmixed}
%\setcopyright{cagov}
%\setcopyright{cagovmixed}


% DOI
%\acmDOI{10.475/123_4}
\acmDOI{none}

% ISBN
%\acmISBN{123-4567-24-567/08/06}
\acmISBN{none}

%Conference
\acmConference[KDD'18]{ACM}{July 2018}{London, UK}
%  \acmYear{1997}
%  \copyrightyear{2016}

\settopmatter{printfolios=true}

%% Prepare for arXiv
% \settopmatter{printacmref=false} % Removes citation information below abstract
% \renewcommand\footnotetextcopyrightpermission[1]{} % removes footnote with conference information in first column
% \pagestyle{plain}


%\acmArticle{4}
%\acmPrice{15.00}

% These commands are optional
%\acmBooktitle{Transactions of the ACM Woodstock conference}
%\editor{Jennifer B. Sartor}
%\editor{Theo D'Hondt}
%\editor{Wolfgang De Meuter}


\begin{document}

\title{Consistent Individualized Feature Attribution for Tree Ensembles}

\author{Scott M. Lundberg, Gabriel G. Erion,  and Su-In Lee}
\affiliation{%
  \institution{University of Washington}
%  \streetaddress{University of Washington}
%   \city{Seattle}
%   \state{Washington}
%   \postcode{98105}
}
\email{{slund1,erion,suinlee}@uw.edu}
%\author{}

% \author{G.K.M. Tobin}
% \authornote{The secretary disavows any knowledge of this author's actions.}
% \affiliation{%
%   \institution{Institute for Clarity in Documentation}
%   \streetaddress{P.O. Box 1212}
%   \city{Dublin}
%   \state{Ohio}
%   \postcode{43017-6221}
% }
% \email{webmaster@marysville-ohio.com}

% \author{Lars Th{\o}rv{\"a}ld}
% \authornote{This author is the
%   one who did all the really hard work.}
% \affiliation{%
%   \institution{The Th{\o}rv{\"a}ld Group}
%   \streetaddress{1 Th{\o}rv{\"a}ld Circle}
%   \city{Hekla}
%   \country{Iceland}}
% \email{larst@affiliation.org}

% \author{Valerie B\'eranger}
% \affiliation{%
%   \institution{Inria Paris-Rocquencourt}
%   \city{Rocquencourt}
%   \country{France}
% }
% \author{Aparna Patel}
% \affiliation{%
%  \institution{Rajiv Gandhi University}
%  \streetaddress{Rono-Hills}
%  \city{Doimukh}
%  \state{Arunachal Pradesh}
%  \country{India}}
% \author{Huifen Chan}
% \affiliation{%
%   \institution{Tsinghua University}
%   \streetaddress{30 Shuangqing Rd}
%   \city{Haidian Qu}
%   \state{Beijing Shi}
%   \country{China}
% }

% \author{Charles Palmer}
% \affiliation{%
%   \institution{Palmer Research Laboratories}
%   \streetaddress{8600 Datapoint Drive}
%   \city{San Antonio}
%   \state{Texas}
%   \postcode{78229}}
% \email{cpalmer@prl.com}

% \author{John Smith}
% \affiliation{\institution{The Th{\o}rv{\"a}ld Group}}
% \email{jsmith@affiliation.org}

% \author{Julius P.~Kumquat}
% \affiliation{\institution{The Kumquat Consortium}}
% \email{jpkumquat@consortium.net}

% The default list of authors is too long for headers.
%\renewcommand{\shortauthors}{B. Trovato et al.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Elaborate the importance of consistency
% Comments on Figure 1

\begin{abstract}
Interpreting predictions from tree ensemble methods such as gradient boosting machines and random forests is important, yet feature attribution for trees is often heuristic and not individualized for each prediction. Here we show that popular feature attribution methods are \emph{inconsistent}, meaning they can lower a feature's assigned importance when the true impact of that feature actually increases. This is a fundamental problem that casts doubt on any comparison between features. To address it we turn to recent applications of game theory and develop fast exact tree solutions for SHAP (\underline{SH}apley \underline{A}dditive ex\underline{P}lanation) values, which are the unique consistent and locally accurate attribution values.  We then extend SHAP values to interaction effects and define {\it SHAP interaction values}. We propose a rich visualization of individualized feature attributions that improves over classic attribution summaries and partial dependence plots, and a unique ``supervised'' clustering (clustering based on feature attributions). We demonstrate better agreement with human intuition through a user study, exponential improvements in run time, improved clustering performance, and better identification of influential features. An implementation of our algorithm has also been merged into XGBoost and LightGBM, see \url{http://github.com/slundberg/shap} for details.

% particularly in domains such as finance or medicine, where each prediction needs a trustworthy explanation.
% Feature importance values are a key part of understanding widely used models such as gradient boosting trees and random forests, so improvements to them have significant practical implications.

% To address this problem we specialize recent work on provably consistent attribution values from game theory 


% Tree ensemble methods such as gradient boosting and random forests are widely used, yet feature importance attribution for them is often heuristic and not individualized for each prediction. Here we show that popular feature attribution methods are \emph{inconsistent}, which means changing the model to rely more on a given feature can actually decrease the attributed importance of that feature. This is a fundamental problem that casts doubt on any comparison between features. 
% To address it we turn to recent applications of game theory and develop fast exact tree solutions for SHAP (\underline{SH}apley \underline{A}dditive ex\underline{P}lanation) values, which are the unique consistent and locally accurate attribution values. We then extend SHAP values to interaction effects and define {\it SHAP interaction values}. We demonstrate better agreement with human intuition through a user study, exponential improvements in run time, improved supervised clustering performance, better identification of influential features, and improved interpretability over classic attribution approaches and partial dependence plots. An implementation of the proposed algorithm has also been merged into XGBoost and LightGBM.

% It is critical in many applications to understand what features are important for a model prediction, and why individual predictions were made. For tree ensemble methods these questions are usually answered by attributing importance values to input features, either globally or for a single prediction. Here we show that current feature attribution methods are \emph{inconsistent}, which means changing the model to rely more on a given feature can actually decrease the importance estimate of that feature.
% %
% To address this problem we develop fast exact solutions for SHAP (\underline{SH}apley \underline{A}dditive ex\underline{P}lanation) values, which were recently shown to be the unique additive feature attribution method based on conditional expectations that is both consistent and locally accurate. We integrate these improvements into the latest version of XGBoost, demonstrate the inconsistencies of current methods, show how using SHAP values results in significantly improved supervised clustering performance, and note better agreement with human intuition through a user study.
% Feature importance values are a key part of understanding widely used models such as gradient boosting trees and random forests, so improvements to them have significant practical implications.
\end{abstract}

\maketitle

%% 1. Individualized vs. global -- We need to explain somewhere early in the Intro.

\section{Introduction}

Understanding why a model made a prediction is important for trust, actionability, accountability, debugging, and many other tasks. To understand predictions from tree ensemble methods, such as gradient boosting machines or random forests, importance values are typically attributed to each input feature. These importance values can be computed either for a single prediction (individualized), or an entire dataset to explain a model's overall behavior (global).

Concerningly, popular current feature attribution methods for tree ensembles are {\it inconsistent}. This means that when a model is changed such that a feature has a higher impact on the model's output, current methods can actually lower the importance of that feature. Inconsistency strikes at the heart of what it means to be a good attribution method, because it prevents the meaningful comparison of attribution values across features. %Relying on inconsistent attribution methods can lead to problems with any task based on comparing one feature attribution to another, such as ranking features, or comparing between alternative models. 
%A feature with a smaller attributed importance might actually be more important than a feature with a higher attributed importance. 
This is because inconsistency implies that a feature with a large attribution value might be less important than another feature with a smaller attribution (see Figure \ref{fig:and_trees} and Section \ref{sec:current}).

% previosuly unappreciated...

% nconsistent prevent the meaningful comparsion between features  means that ranking features or This casts doubt on any useor of features, or any comparison between models 

% fundementally diaffects a large number of users, since tree ensemble methods are widely applied in research and industry.

To address this problem we turn to the recently proposed SHAP (SHapley Additive exPlanation) values \cite{lundberg2017unified}, which are based on a unification of ideas from game theory \cite{vstrumbelj2014explaining} and local explanations \cite{ribeiro2016should}. Here we show that by connecting tree ensemble feature attribution methods with the class of {\it additive feature attribution methods} \cite{lundberg2017unified} we can motivate SHAP values as the only possible consistent feature attribution method with several desirable properties.

SHAP values are theoretically optimal, but like other model agnostic feature attribution methods \cite{baehrens2010explain,ribeiro2016should,vstrumbelj2014explaining,datta2016algorithmic}, they can be challenging to compute. To solve this we derive an algorithm for tree ensembles that reduces the complexity of computing exact SHAP values from $O(T L2^M)$ to $O(T LD^2)$ where $T$ is the number of trees, $L$ is the maximum number of leaves in any tree, $M$ is the number of features, and $D$ is the maximum depth of any tree. This exponential reduction in complexity allows predictions from previously intractable models with thousands of trees and features to now be explained in a fraction of a second. Entire datasets can now be explained, which enables new alternatives to traditional partial dependence plots and feature importance plots \cite{friedman2001elements}, which we term {\it SHAP dependence plots} and {\it SHAP summary plots}, respectively. % mention supervised clustering

Current attribution methods cannot directly represent interactions, but must divide the impact of an interaction among each feature. To directly capture pairwise interaction effects we propose {\it SHAP interaction values}; an extension of SHAP values based on the Shapley interaction index from game theory \cite{fujimoto2006axiomatic}. SHAP interaction values bring the benefits of guaranteed consistency to explanations of interaction effects for individual predictions.

In what follows we first discuss current tree feature attribution methods and their inconsistencies. We then introduce SHAP values as the only possible consistent and locally accurate attributions, present Tree SHAP as a high speed algorithm for estimating SHAP values of tree ensembles, then extend this to SHAP interaction values. We use user study data, computational performance, influential feature identification, and supervised clustering to compare with previous methods. Finally, we illustrate SHAP dependence plots and SHAP summary plots with XGBoost and NHANES I national health study data \cite{miller1973plan}.

\begin{figure*}
  \centering
  \includegraphics[width=0.99\textwidth]{figures/and_trees}
  \caption[Simple example]{Two simple tree models that demonstrate inconsistencies in the Saabas, gain, and split count attribution methods: The Cough feature has a larger impact in Model B than Model A, but is attributed less importance in Model B. Similarly, the Cough feature has a larger impact than Fever in Model B, yet is attributed less importance. The individualized attributions explain a single prediction of the model (when both Cough and Fever are Yes) by allocating the difference between the expected value of the model's output (20 for Model A, 25 for Model B) and the current output (80 for Model A, 90 for Model B). The global attributions represent the overall importance of a feature in the model. Without consistency it is impossible to reliably compare feature attribution values.
  %change in the mean squared error over the whole dataset between when no features are used and all features are used. All calculations assume a dataset (typically a training dataset) perfectly matching the model and evenly spread among all leaves. 
  %Section \ref{sec:current} describes the standard ``path" methods, while Section \ref{sec:shap} describes the SHAP values and their interpretation.
  }
  \label{fig:and_trees}
\end{figure*}

\section{Inconsistencies in current feature attribution methods}
\label{sec:current}

Tree ensemble implementations in popular packages such as XGBoost \cite{chen2016xgboost}, scikit-learn \cite{pedregosa2011scikit}, and the {\it gbm} R package \cite{ridgeway2010generalized} allow a user to compute a measure of feature importance. These values are meant to summarize a complicated ensemble model and provide insight into what features drive the model's prediction.

{\it Global} feature importance values are calculated for an entire dataset (i.e., for all samples) in three primary ways:
\begin{enumerate}
\item Gain: A classic approach to feature importance introduced by Breiman et al. in 1984 \cite{breiman1984classification} is based on gain. Gain is the total reduction of loss or impurity contributed by all splits for a given feature. Though its motivation is largely heuristic \cite{friedman2001elements}, gain is widely used as the basis for feature selection methods \cite{sandri2008bias, irrthum2010inferring, chebrolu2005feature}.
\item Split Count: A second common approach %measure of global feature importance 
is simply to count how many times a feature is used to split \cite{chen2016xgboost}. Since feature splits are chosen to be the most informative, this can represent a feature's importance.
\item Permutation: A third common approach %to feature importance for trees 
is to randomly permute the values of a feature in the test set and then observe the change in the model's error. If a feature's value is important then permuting it should create a large increase in the model's error. Different choices about the method of feature value permutation lead to variations of this basic approach \cite{strobl2008conditional,ishwaran2007variable, auret2011empirical,diaz2006gene,rodenburg2008framework}.
\end{enumerate}

{\it Individualized} methods that compute feature importance values for a single prediction are less established for trees. While model agnostic individualized explanation methods \cite{lundberg2017unified,baehrens2010explain,ribeiro2016should,vstrumbelj2014explaining,datta2016algorithmic} can be applied to trees \cite{lundberg2017explainable}, they are significantly slower than tree-specific methods and have sampling variability (see Section \ref{sec:experiments} for a computational comparison, or \cite{lundberg2017unified} for an overview). The only current tree-specific individualized explanation method we are aware of is by Sabbas \cite{path_blog}. The Saabas method is similar to the classic dataset-level gain method, but instead of measuring the reduction of loss, it measures the change in the model's expected output. It proceeds by comparing the expected value of the model output at the root of the tree with the expected output of the sub-tree rooted at the child node followed by the decision path of the current input. The difference between these expectations is then attributed to the feature split on at the root node. By repeating this process recursively the method allocates the difference between the expected model output and the current output among the features on the decision path.

\begin{figure*}
  \centering
  \includegraphics[width=0.9\textwidth]{figures/number_line}
  \caption{SHAP (\underline{SH}apley \underline{A}dditive ex\underline{P}lanation) values explain the output of a function $f$ as a sum of the effects $\phi_i$ of each feature being introduced into a conditional expectation. Importantly, for non-linear functions the order in which features are introduced matters. SHAP values result from averaging over all possible orderings. Proofs from game theory show this is the only possible consistent approach where $\sum_{i=0}^M \phi_i = f(x)$. In contrast, the only current individualized feature attribution method for trees satisfies the summation, but is inconsistent because it only considers a single ordering \cite{path_blog}.}
  \label{fig:number_line}
\end{figure*}

%$Both current feature attribution methods described above only consider the effect of splits along the decision path, so we will term them {\it path} methods. 
\emph{Unfortunately, the feature importance values from the gain, split count, and Saabas methods are all inconsistent.} This means that a model can change such that it relies more on a given feature, yet the importance estimate assigned to that feature decreases. Of the methods we consider, 
only SHAP values and permutation-based methods are consistent.
%A simple two level regression tree demonstrates the inconsistency of Of the global feature attribution methods above, only those based on permutations of feature values are consistent, while those based on variations of gain or split counts are inconsistent. The only individualized feature attribution method, Saabas, is also inconsistent. To illustrate the inconsistency is (Figure \ref{fig:and_trees})
Figure \ref{fig:and_trees} shows the result of applying all these methods to two simple regression trees.\footnote{For clarity we rounded small values in Figure \ref{fig:and_trees}. These small values are why the lower left splits in both models were not pruned during training.} For the global calculations we assume an equal number of dataset points fall in each leaf, and the label of those points is exactly equal to the prediction of the leaf. Model~A represents a simple AND function, while Model~B represents the same AND function but with an additional increase in the predicted value when Cough is ``Yes''. Note that because Cough is now more important it gets split on first in Model~B.

Individualized feature attribution is represented by Tree~SHAP and Sabbas for the input Fever=Yes and Cough=Yes. Both methods allocate the difference between the current model output and the expected model output among the input features ($80 - 20$ for Model~A). But the SHAP values are guaranteed to reflect the importance of the feature (see Section~\ref{sec:shap}), while the Saabas values can give erroneous results, such as a larger attribution to Fever than to Cough in Model~B.

Global feature attribution is represented by four methods: the mean magnitude of the SHAP values, gain, split count, and feature permutation. Only the mean SHAP value magnitude and permutation correctly give Cough more importance than Fever in Model~B. This means gain and split count are not reliable measures of global feature importance, which is important to note given their widespread use.

% Cough, each of which divide the change prediction

% The point of Figure \ref{fig:and_trees} is the comparison of feature attributions between Model A and Model B, where it is clear that Cough has a larger impact on Model B than Model A. As shown in the bar chart the Saabas, gain, and split count approaches all give a lower attribution to the feature Cough in Model B than Model A, even though Cough is clearly more important in Model B. This is inconsistency, and it prevents the meaningful comparison of features.

% highlighted below each tree, we can see that current path methods (as well as the simple split count metric) are inconsistent because they allocate less importance to Cough in B, even though Cough has a larger impact on the output of the tree in B. The ``output" task explains the change in model output from the expected value to the current predicted value given Fever=Yes and Cough=Yes; it explains a single prediction. The ``gain" explains the reduction in mean squared error contributed by each feature; it explains the model over the whole dataset described in the previous paragraph. In contrast to current approaches, the SHAP values described below are consistent, even when the order in which features appear in the tree changes.


\subsection{SHAP values as the only consistent and locally accurate individualized feature attributions}
\label{sec:shap}

It was recently noted that many current methods for interpreting individual machine learning model predictions fall into the class of {\it additive feature attribution methods} \cite{lundberg2017unified}. This class covers methods that explain a model's output as a sum of real values attributed to each input feature.

\begin{definition}
\label{def:additive}
{\bf Additive feature attribution methods} have an explanation model $g$ that is a linear function of binary variables:
\begin{equation}
\label{eq:additive_fa}
g(z') = \phi_0 + \sum_{i = 1}^M \phi_i z_i',
\end{equation}
where $z' \in \{0,1\}^M$, $M$ is the number of input features, and $\phi_i \in \mathbb{R}$.
\end{definition}

The $z'_i$ variables typically represent a feature being observed ($z'_i = 1$) or unknown ($z'_i = 0$), and the $\phi_i$'s are the feature attribution values.

As previously described in \citeauthor{lundberg2017unified} (2017), an important property of the class of additive feature attribution methods is that there is a single unique solution in this class with three desirable properties: {\it local accuracy}, {\it missingness}, and {\it consistency}. Local accuracy states that the sum of the feature attributions is equal to the output of the function we are seeking to explain. Missingness states that features that are already missing (such that $z'_i = 0$) are attributed no importance. Consistency states that changing a model so a feature has a larger impact on the model will never decrease the attribution assigned to that feature.

%The only previous individualized feature attribution method for trees, the Saabas method, satisfies both local accuracy and missingness, but fails to satisfy consistency.

%which is to our knowledge t  we are aware Current feature attribution methods for tree ensemble satisfy local accuracy and missingness, but fail to satisfy consistency. 

Note that in order to evaluate the effect missing features have on a model $f$, it is necessary to define a mapping $h_x$ that maps between a binary pattern of missing features represented by $z'$ and the original function input space. Given such a mapping we can evaluate $f(h_x(z'))$ and so calculate the effect of observing or not observing a feature (by setting $z'_i = 1$ or $z'_i = 0$).

To compute SHAP values we define $f_x(S) = f(h_x(z')) = E[f(x) \mid x_S]$ where $S$ is the set of non-zero indexes in $z'$ (Figure \ref{fig:number_line}), and $E[f(x) \mid x_S]$ is the expected value of the function conditioned on a subset $S$ of the input features. SHAP values combine these conditional expectations with the classic Shapley values from game theory to attribute $\phi_i$ values to each feature:

\begin{equation}
\label{eq:shapley}
\phi_i = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!(M - |S| -1)!}{M!} \left [ f_x(S \cup \{i\}) - f_x(S) \right ],
\end{equation}

\noindent where $N$ is the set of all input features.

As shown in \citeauthor{lundberg2017unified} (2017), the above method is the only possible consistent, locally accurate method that obeys the missingness property and uses conditional dependence to measure missingness \cite{lundberg2017unified}. This is strong motivation to use SHAP values for tree ensemble feature attribution, particularly since the only previous individualized feature attribution method for trees, the Saabas method, satisfies both local accuracy and missingness using conditional dependence, but fails to satisfy consistency. This means that SHAP values provide a strict theoretical improvement by eliminating significant consistency problems (Figure~\ref{fig:and_trees}).

\section{Tree SHAP: Fast SHAP value computation for trees}
\label{sec:tree_shap}

Despite the compelling theoretical advantages of SHAP values, their practical use is hindered by two problems:

\begin{enumerate}
\item The challenge of estimating $E[f(x) \mid x_S]$ efficiently.
\item The exponential complexity of Equation \ref{eq:shapley}.
\end{enumerate}

Here we focus on tree models and propose fast SHAP value estimation methods specific to trees and ensembles of trees. We start by defining a slow but straightforward algorithm, then present the much faster and more complex Tree~SHAP algorithm.

\subsection{Estimating SHAP values directly in $O(TL2^M)$ time}
\label{sec:shap_direct}

If we ignore computational complexity then we can compute the SHAP values for a tree by estimating $E[f(x) \mid x_S]$ and then using Equation \ref{eq:shapley} where $f_x(S) = E[f(x) \mid x_S]$. For a tree model $E[f(x) \mid x_S]$ can be estimated recursively using Algorithm \ref{alg:exp_value}, where $v$ is a vector of node values, which takes the value $internal$ for internal nodes. The vectors $a$ and $b$ represent the left and right node indexes for each internal node. The vector $t$ contains the thresholds for each internal node, and $d$ is a vector of indexes of the features used for splitting in internal nodes. The vector $r$ represents the cover of each node (i.e., how many data samples fall in that sub-tree). The weight $w$ measures what proportion of the training samples matching the conditioning set $S$ fall into each leaf.

\begin{algorithm}
\caption{Estimating $E[f(x) \mid x_S]$ \label{alg:exp_value}}
\begin{algorithmic}
\Procedure{EXPVALUE}{$x$, $S$, $tree = \{v, a, b, t, r, d\}$}
\Procedure{G}{$j$, $w$}
  \If{$v_j \ne internal$}
    \State \Return{$w \cdot v_j$}
  \Else
    \If{$d_j \in S$}
      \State \Return \Call{G}{$a_j$, $w$} {\bf if} $x_{d_j} \le t_j$ {\bf else} \Call{G}{$b_j$, $w$}
    \Else
    \State \Return \Call{G}{$a_j$, $w r_{a_j} / r_j$} + \Call{G}{$b_j$, $w r_{b_j} / r_j$}
    \EndIf
    
  \EndIf
\EndProcedure
\State \Return \Call{G}{$1$, $1$}
\EndProcedure
\end{algorithmic}
\end{algorithm}

\subsection{Estimating SHAP values in $O(TLD^2)$ time}
\label{sec:shap_fast}

Here we propose a novel algorithm to calculate the same values as above, but in polynomial time instead of exponential time. Specifically, we propose an algorithm that runs in $O(TLD^2)$ time and $O(D^2 + M)$ memory, where for balanced trees the depth becomes $D = \log L$. Recall $T$ is the number of trees, $L$ is the maximum number of leaves in any tree, and $M$ is the number of features.

%\textcolor{red}{runs in $O(TL \log^2 L)$ [SIL: log superscript 2??]} time with $O(\log^2 L + M)$ memory for balanced trees, and

The intuition of the polynomial time algorithm is to recursively keep track of what proportion of all possible subsets flow down into each of the leaves of the tree. 
This is similar to running Algorithm \ref{alg:exp_value} simultaneously for all $2^M$ subsets $S$ in Equation \ref{eq:shapley}. It may seem reasonable to simply keep track of how many subsets (weighted by the cover splitting of Algorithm \ref{alg:exp_value}) pass down each branch of the tree. However, this combines subsets of different sizes and so prevents the proper weighting of these subsets, since the weights in Equation \ref{eq:shapley} depend on $|S|$. To address this we keep track of each possible subset size during the recursion. The {\it EXTEND} method in Algorithm \ref{alg:tree_shap} grows all these subsets according to a given fraction of ones and zeros, while the {\it UNWIND} method reverses this process and is commutative with {\it EXTEND}. The {\it EXTEND} method is used as we descend the tree. The {\it UNWIND} method is used to undo previous extensions when we split on the same feature twice, and to undo each extension of the path inside a leaf to compute weights for each feature in the path.

\begin{algorithm}
\caption{Tree SHAP \label{alg:tree_shap}}
\begin{algorithmic}
\Procedure{TS}{$x$, $tree = \{v, a, b, t, r, d\}$}
\State $\phi = \textrm{array of $len(x)$ zeros}$
\Procedure{RECURSE}{$j$, $m$, $p_z$, $p_o$, $p_i$}
  \State $m =~$\Call{EXTEND}{$m$, $p_z$, $p_o$, $p_i$}
  \If{$v_j \ne internal$}
    \For{$i \gets 2 \textrm{ to } len(m)$}
      \State $w = sum(\Call{UNWIND}{m, i}.w)$
      \State $\phi_{m_i} = \phi_{m_i} + w(m_i.o - m_i.z) v_j$
    \EndFor
  \Else
    \State $h,c = x_{d_j} \le t_j~?~(a_j,b_j) : (b_j,a_j)$
    \State $i_z = i_o = 1$
    \State $k = \Call{FINDFIRST}{m.d, d_j}$
    \If{$k \ne \textrm{nothing}$}
      \State $i_z,i_o = (m_k.z,m_k.o)$
      \State $m = \Call{UNWIND}{m, k}$
    \EndIf
    \State \Call{RECURSE}{$h$, $m$, $i_z r_h/r_j$, $i_o$, $d_j$}
    \State \Call{RECURSE}{$c$, $m$, $i_z r_c/r_j$, $0$, $d_j$}
  \EndIf
\EndProcedure
\Procedure{EXTEND}{$m$, $p_z$, $p_o$, $p_i$}
\State $l = len(m)$
\State $m = copy(m)$
\State $m_{l+1}.(d,z,o,w) = (p_i, p_z, p_o, l = 0~?~1:0)$
\For{$i \gets l-1 \textrm{ to } 1$}
  \State $m_{i+1}.w = m_{i+1}.w + p_o m_i.w(i/l)$
  \State $m_i.w =  p_z m_i.w[(l-i)/l]$
\EndFor
\State \Return m
\EndProcedure
\Procedure{UNWIND}{$m$, $i$}
\State $l = len(m)$
\State $n = m_l.w$
\State $m = copy(m_{1...l-1})$
\For{$j \gets l-1 \textrm{ to } 1$}
  \If{$m_i.o \ne 0$}
    \State $t = m_j.w$
    \State $m_j.w = n \cdot l/(j \cdot m_i.o)$
    \State $n = t - m_j.w \cdot m_i.z ((l-j)/l)$
  \Else
    \State $m_j.w = (m_j.w \cdot l)/(m_i.z (l-j))$
  \EndIf
\EndFor
\For{$j \gets i \textrm{ to } l-1$}
  \State $m_j.(d,z,o) = m_{j+1}.(d,z,o)$
\EndFor
\State \Return m
\EndProcedure
\State \Call{RECURSE}{$1$, $[]$, $1$, $1$, $0$}
\State \Return $\phi$
\EndProcedure
\end{algorithmic}
\end{algorithm}

In Algorithm \ref{alg:tree_shap}, $m$ is the path of unique features we have split on so far, and contains four attributes: $d$ the feature index, $z$ the fraction of ``zero'' paths (where this feature is not in the set $S$) that flow through this branch, $o$ the fraction of ``one'' paths (where this feature is in the set $S$) that flow through this branch, and $w$ which is used to hold the proportion of sets of a given cardinality that are present. We use the dot notation to access these members, and for the whole vector $m.d$ represents a vector of all the feature indexes.

Algorithm \ref{alg:tree_shap} reduces the computational complexity of exact SHAP value computation from exponential to low order polynomial for trees and sums of trees (since the SHAP values of a sum of two functions is the sum of the original functions' SHAP values).%\footnote{For code see \url{https://github.com/slundberg/shap}}

%% HERE is where Su-In suggested we put a paragraph about the SHAP plots...I have not figured out how to word it yet.

% \begin{figure*}
%   \centering
%   \includegraphics[width=1.0\textwidth]{figures/clustering}
%   \caption{SHAP feature attributions produce better clusters than standard path attributions for supervised clustering of 518 participants in an Alzheimer's research study. An XGBoost model with 300 trees of max depth six was trained on 200 gene expression module features using a shrinkage factor of $\eta = 0.01$. This model was then used to predict the CERAD cognitive score of each participant. Each prediction was explained, and then clustered using hierarchical agglomerative clustering (imagine a dendrogram joining the samples above each plot). Red feature attributions push the score higher, while blue feature attributions push the score lower. A) The clusters formed with standard ``path" explanations from XGBoost. B) Clusters using our Tree SHAP XGBoost implementation.}
%   \label{fig:clustering}
% \end{figure*}

%\section{SHAP Interaction Values}

% \textcolor{red}{\subsection{Applications}
% Consistency enables ..
% }

\section{SHAP Interaction Values}

Feature attributions are typically allocated among the input features, one for each feature, but we can gain additional insight by separating \emph{interaction effects} from main effects. If we consider pairwise interactions this leads to a matrix of attribution values representing the impact of all pairs of features on a given model prediction. Since SHAP values are based on classic Shapley values from game theory, a natural extension to interaction effects can be obtained though the more modern Shapley interaction index \cite{fujimoto2006axiomatic}:

\begin{equation}
\label{eq:shapley_interactions}
\Phi_{i,j} = \sum_{S \subseteq N \setminus \{i,j\}} \frac{|S|!(M - |S| - 2)!}{2(M - 1)!} \nabla_{ij}(S),
\end{equation}

\noindent when $i \ne j$, and
\begin{align}
\label{eq:shapley_interaction_grad}
\nabla_{ij}(S) &= f_x(S \cup \{i,j\}) - f_x(S \cup \{i\}) - f_x(S \cup \{j\}) + f_x(S) \\
&= f_x(S \cup \{i,j\}) - f_x(S \cup \{j\}) - [f_x(S \cup \{i\}) - f_x(S)]. \label{eq:shapley_interaction_grad2}
\end{align}

\noindent In Equation~\ref{eq:shapley_interactions} the SHAP interaction value between feature $i$ and feature $j$ is split equally between each feature so $\Phi_{i,j} = \Phi_{j,i}$ and the total interaction effect is $\Phi_{i,j} + \Phi_{j,i}$. The main effects for a prediction can then be defined as the difference between the SHAP value and the SHAP interaction values for a feature:

\begin{align}
\label{eq:shapley_interaction_main}
\Phi_{i,i} &= \phi_i - \sum_{j \ne i} \Phi_{i,j}.
\end{align}

These SHAP interaction values follow from similar axioms as SHAP values, and allow the separate consideration of main and interaction effects for individual model predictions. This separation can uncover important interactions captured by tree ensembles that might otherwise be missed (Figure \ref{fig:nhanes_sbp_main_and_age_interaction} in Section~\ref{sec:shap_plots}).

While SHAP interaction values can be computed directly from Equation~\ref{eq:shapley_interactions}, we can leverage Algorithm~\ref{alg:tree_shap} to drastically reduce their computational cost for tree models. As highlighted in Equation~\ref{eq:shapley_interaction_grad2} SHAP interaction values can be interpreted as the difference between the SHAP values for feature $i$ when feature $j$ is present and the SHAP values for feature $i$ when feature $j$ is absent. This allows us to use Algorithm~\ref{alg:tree_shap} twice, once while ignoring feature $j$ as fixed to present, and once with feature $j$ absent. This leads to a run time of $O(TMLD^2)$, since we repeat the process for each feature. Note that even though this computational approach does not seem to directly enforce symmetry, the resulting $\Phi$ matrix is always symmetric.


\section{Experiments and Applications}

We compare Tree SHAP and SHAP interaction values with previous methods through both traditional metrics and three new applications we propose for individualized feature attributions: supervised clustering, SHAP summary plots, and SHAP dependence plots.\footnote{Jupyter notebooks to compute all results are available at \url{http://github.com/slundberg/shap/notebooks/tree_shap_paper}}

\begin{figure}
  \centering
  \includegraphics[width=0.38\textwidth]{figures/tree_max}
  \caption{Feature attribution values from 34 participants shown the tree from Model A in Figure \ref{fig:and_trees}. The first number represents the allocation to the Fever feature, while the second represents the allocation to the Cough feature. Participants from Amazon Mechanical Turk were not selected for machine learning expertise. No constraints were placed on the feature attribution values users entered.}
  \label{fig:tree_max}
\end{figure}

\begin{figure*}
  \centering
  \includegraphics[width=1.0\textwidth]{figures/census_explanation2}
  \caption{Supervised clustering with SHAP feature attributions in the UCI census dataset identifies among 2,000 individuals distinct subgroups of people that share similar reasons for making money. An XGBoost model with 500 trees of max depth six was trained on demographic data using a shrinkage factor of $\eta = 0.005$. This model was then used to predict the log odds that each person makes $\ge\$50K$. Each prediction was explained using Tree SHAP, and then clustered using hierarchical agglomerative clustering (imagine a dendrogram above the plot joining the samples). Red feature attributions push the score higher, while blue feature attributions push the score lower (as in Figure \ref{fig:number_line} but rotated 90$^\circ$). A few of the noticeable subgroups are annotated with the features that define them.}
  \label{fig:census_explanation}
\end{figure*}

\subsection{Agreement with Human Intuition}
\label{sec:user_study}

To validate that the SHAP values in Model A of Figure \ref{fig:and_trees} are the most natural assignment of credit we ran a user study to measure people's intuitive feature attribution values. Model A's tree was shown to participants and said to represent risk for a certain disease. They were told that when a given person was found to have both a cough and fever their risk went up from the prior risk of 20 (the expected value of risk) to a risk of 80. Participants were then asked to apportion the 60 point change in risk among the Cough and Fever features as they saw best.

Figure \ref{fig:tree_max} presents the results of the user study for Model A. The equal distribution of credit used by SHAP values was found to be the most intuitive. A smaller number of participants preferred to give greater weight to the first feature to be split on (Fever), while still fewer followed the allocation of the Saabas method and gave greater weight to the second feature split on (Cough).

\subsection{Computational Performance}
\label{sec:run_time}

Figure \ref{fig:runtime} demonstrates the significant run time improvement provided by Algorithm 2. Problems that were previously intractable for exact computation are now inexpensive. An XGBoost model with 1,000 depth 10 trees over 100 input features can now be explained in 0.08 seconds.

\begin{figure}
  \centering
  \includegraphics[width=0.42\textwidth]{figures/runtime}
  \caption{Runtime improvement of Algorithm 2 over using Equation \ref{eq:shapley} and Algorithm 1. An XGBoost model with 50 trees was trained using an equally increasing number of input features and max tree depths. The time to explain one input vector is reported.}
  \label{fig:runtime}
\end{figure}


\subsection{Supervised Clustering}
\label{sec:experiments}
%While sound theoretical consistency guarantees are important, it is also good to demonstrate SHAP values bring value to tasks that utilize feature attributions. 
One intriguing application enabled by individualized feature attributions is what we term ``supervised clustering,'' where instead of using an unsupervised clustering method directly on the data features, you run clustering on the feature attributions.

Supervised clustering naturally handles one of the most challenging problems in unsupervised clustering: determining feature weightings (or equivalently, determining a distance metric). Many times we want to cluster data using features with very different units. 
Features may be in dollars, meters, unit-less scores, etc. but whenever we use them as dimensions in a single multidimensional space it forces any distance metric to compare the relative importance of a change in different units (such as dollars vs. meters). Even if all our inputs are in the same units, often some features are more important than others. Supervised clustering uses feature attributions to naturally convert all the input features into values with the same units as the model output. This means that a unit change in any of the feature attributions is comparable to a unit change in any other feature attribution. It also means that fluctuations in the feature values only effect the clustering if those fluctuations have an impact on the outcome of interest.



Here we demonstrate the use of supervised clustering on the classic UCI census dataset \cite{Lichman2013}. For this dataset the goal is to predict from basic demographic data if a person is likely to make more than \$50K annually.
% compare feature attribution methods by applying supervised clustering to disease sub-typing, an area where unsupervised clustering has contributed to important discoveries. The goal of disease sub-typing is to identify subgroups of patients that have similar mechanisms of disease (similar reasons they are sick). Here we consider Alzheimer's disease where the predicted outcome is the CERAD cognitive score \cite{mirra1991consortium}, and the features are gene expression modules \cite{celik2014efficient}.
By representing the positive feature attributions as red bars and the negative feature attributions as blue bars (as in Figure \ref{fig:number_line}), we can stack them against each other to visually represent the model output as their sum. Figure \ref{fig:census_explanation} does this vertically for predictions from 2,000 people from the census dataset. The explanations for each person are stacked horizontally according the leaf order of a hierarchical clustering of the SHAP values. This groups people with similar reasons for a predicted outcome together. The formation of distinct subgroups of people demonstrates the power of supervised clustering to identify groups that share common factors related to income level.

\begin{figure}[t!h]
  \centering
  \includegraphics[width=0.4\textwidth]{figures/cluster_plots}
  \caption{A quantitative measure of supervised clustering performance. If all samples are placed in their own group, and each group predicts the mean value of the group, then the $R^2$ value (the proportion of model output variance explained) will be $1$. If groups are then merged one-by-one the $R^2$ will decline until when there is only a single group it will be $0$. Hierarchical clusterings that well separate the model output value will retain a high $R^2$ longer during the merging process. Here supervised clustering with SHAP values outperformed the Sabbas method in both (A) the census data clustering shown in Figure \ref{fig:census_explanation}, and (B) a clustering from gene-based predictions of Alzheimer's cognitive scores.}
  \label{fig:cluster_plots}
\end{figure}

One way to quantify the improvement provided by SHAP values over the heuristic Saabas attributions is by examining how well supervised clustering based on each method explains the variance of the model output (note global feature attributions are not considered since they do not enable this type of supervised clustering). If feature attribution values well-represent the model then supervised clustering groups will have similar function outputs. Since hierarchical clusterings encode many possible groupings, we plot in Figure \ref{fig:cluster_plots} the change in the $R^2$ value as the number of groups shrinks from one group per sample ($R^2 = 1$) to a single group ($R^2 = 0$). For the census dataset, groupings based on SHAP values outperform those from Saabas values (Figure \ref{fig:cluster_plots}A). For a dataset based on cognitive scores for Alzheimer's disease SHAP values significantly outperform Saabas values (Figure \ref{fig:cluster_plots}B). This second dataset contains 200 gene expression module levels \cite{celik2014efficient} as features and CERAD cognitive scores as labels \cite{mirra1991consortium}.


% \begin{figure}
%   \centering
%   \includegraphics[width=0.4\textwidth]{figures/auc}
%   \caption{A quantitative performance measure of the clusterings shown in Figure \ref{fig:clustering}. If all 518 samples are placed in their own group, and each group predicts the mean value of the group, then the $R^2$ value (the proportion of outcome variance explained) will be $1$. If groups are then merged one-by-one the $R^2$ will decline until when there is only a single group it will be $0$. Hierarchical clusterings that well separate the outcome value will retain a high $R^2$ longer during the merging process. Here unsupervised clustering did no better than random, supervised clustering with the XGBoost ``path" method did significantly better, and SHAP values significantly better still.}
%   \label{fig:auc}
% \end{figure}

\subsection{Identification of Influential Features}

Feature attribution values are commonly used to identify which features influenced a model's prediction the most. To compare methods, the change in a model's prediction can be computed when the most influential feature is perturbed. Figure~\ref{fig:sentiment_perturbation} shows the result of this experiment on a sentiment analysis model of airline tweets \cite{airline_tweets}. An XGBoost model with 50 trees of maximum depth 30 was trained on 11,712 tweets with 1,686 bag-of-words features. Each tweet had a sentiment score label between -1 (negative) and 1 (positive). The predictions of the XGBoost model were then explained for 2,928 test tweets. For each method we choose the most influential negative feature and replaced it with the value of the same feature in another random tweet from the training set (this is designed to mimic the feature being unknown). The new input is then re-run through the model to produce an updated output. If the chosen feature significantly lowered the model output, then the updated model output should be higher than the original. By tracking the total change in model output as we progress through the test tweets we observe that SHAP values best identify the most influential negative feature. Since global methods only select a single feature for the whole dataset we only replaced this feature when it would likely increase the sentiment score (for gain and permutation this meant randomly replacing the ``thank'' feature when it was missing, for split count it was the word ``to''). 

% \begin{figure}
%   \centering
%   \includegraphics[width=0.52\textwidth]{figures/sentiment_summary.pdf}
%   \caption{Runtime improvement of Algorithm 2 over using Equation \ref{eq:shapley} and Algorithm 1. An XGBoost model with 50 trees was trained using an equally increasing number of input features and max tree depths. The time to explain one input vector is reported.}
%   \label{fig:runtime}
% \end{figure}

\begin{figure}
  \centering
  \includegraphics[width=0.45\textwidth]{figures/sentiment_perturbation.pdf}
  \caption{The total increase in a sentiment model's output when the most negative feature is replaced. Five different attribution methods were used to determine the most negative feature for each sample. The higher the total increase in model output, the more accurate the attribution method was at identifying the most influential negative feature.}
  \label{fig:sentiment_perturbation}
\end{figure}

\subsection{SHAP Plots}
\label{sec:shap_plots}

Plotting the impact of features in a tree ensemble model is typically done with a bar chart to represent global feature importance, or a partial dependence plot to represent the effect of changing a single feature \cite{friedman2001elements}. However, since SHAP values are individualized feature attributions, unique to every prediction, they enable new, richer visual representations. 
{\it SHAP summary plots} replace typical bar charts of global feature importance, and {\it SHAP dependence plots} provide an alternative to partial dependence plots that better capture interaction effects.

To explore these visualizations we trained an XGBoost Cox proportional hazards model on survival data from the classic NHANES I dataset \cite{miller1973plan} using the NHANES I Epidemiologic Followup Study \cite{cox1997plan}. After selection for the presence of basic blood test data we obtained data for 9,932 individuals followed for up to 20 years after baseline data collection for mortality. Based on a 80/20 train/test split we chose to use 7,000 trees of maximum depth 3, $\eta = 0.001$, and $50\%$ instance sub-sampling. We then used these parameters and trained on all individuals to generate the final model.

\subsubsection{SHAP Summary Plots}

\begin{figure}
  \centering
  \includegraphics[width=0.48\textwidth]{figures/nhanes_summary.pdf}
  \caption{SHAP summary plot of a 14 feature XGBoost survival model on 20 year mortality followup data from NHANES I \cite{miller1973plan}. The higher the SHAP value of a feature, the higher your log odds of death in this Cox hazards model. Every individual in the dataset is run through the model and a dot is created for each feature attribution value, so one person gets one dot on each feature's line. Dot's are colored by the feature's value for that person and pile up vertically to show density.}
  \label{fig:nhanes_summary}
\end{figure}

Standard feature importance bar charts give a notion of relative importance in the training dataset, but they do not represent the range and distribution of impacts that feature has on the model's output, and how the feature's value relates to it's impact. SHAP summary plots leverage individualized feature attributions to convey all these aspects of a feature's importance while remaining visually concise (Figure~\ref{fig:nhanes_summary}). Features are first sorted by their global impact $\sum_{j=1}^N|\phi_i^{(j)}|$, then dots representing the SHAP values $\phi_i^{(j)}$ are plotted horizontally, stacking vertically when they run out of space. This vertical stacking creates an effect similar to violin plots but without an arbitrary smoothing kernel width. Each dot is colored by the value of that feature, from low (blue) to high (red). If the impact of the feature on the model's output varies smoothly as its value changes then this coloring will also have a smooth gradation. In Figure~\ref{fig:nhanes_summary} we see (unsurprisingly) that age at baseline is the most important risk factor for death over the next 20 years. The density of the age plot shows how common different ages are in the dataset, and the coloring shows a smooth increase in the model's output (a log odds ratio) as age increases. In contrast to age, systolic blood pressure only has a large impact for a minority of people with high blood pressure. The general trend of long tails reaching to the right, but not to the left, means that extreme values of these measurements can significantly raise your risk of death, but cannot significantly lower your risk.

\subsubsection{SHAP Dependence Plots}

%Partial dependence plots are a widely used approach for interpreting tree ensemble methods that do not directly rely on feature attributions. 
As described in Equation 10.47 of \citeauthor{friedman2001elements} (2001), partial dependence plots represent the expected output of a model when the value of a specific variable (or group of variables) is fixed. The values of the fixed variables are varied and the resulting expected model output is plotted. Plotting how the expected output of a function changes as we change a feature helps explain how the model depends on that feature.

SHAP values can be used to create a rich alternative to partial dependence plots, which we term SHAP dependence plots. SHAP dependence plots use the SHAP value of a feature for the y-axis and the value of the feature for the x-axis. By plotting these values for many individuals from the dataset we can see how the feature's attributed importance changes as its value varies (Figure~\ref{fig:nhanes_sbp}). While standard partial dependence plots only produce lines, SHAP dependence plots capture vertical dispersion due to interaction effects in the model. These effects can be visualized by coloring each dot with the value of an interacting feature. In Figure~\ref{fig:nhanes_sbp} coloring by age shows that high blood pressure is more alarming when you are young. Presumably because it is both less surprising as you age, and possibly because it takes time for high blood pressure to lead to fatal complications.

\begin{figure}
  \centering
  \includegraphics[width=0.45\textwidth]{figures/nhanes_sbp.pdf}
  \caption{Each dot is a person. The x-axis is their systolic blood pressure and the y-axis is the SHAP value attributed to their systolic blood pressure. Higher SHAP values represent higher risk of death due to systolic blood pressure. Coloring each dot by the person's age reveals that high blood pressure is more concerning to the model when you are young (this represents an interaction effect).}
  \label{fig:nhanes_sbp}
\end{figure}

Combining SHAP dependence plots with SHAP interaction values can reveal global interaction patterns. Figure~\ref{fig:nhanes_sbp_main_and_age_interaction}A plots the SHAP main effect value for systolic blood pressure. Since SHAP main effect values represents the impact of systolic blood pressure after all interaction effects have been removed (Equation~\ref{eq:shapley_interaction_main}), there is very little vertical dispersion in Figure \ref{fig:nhanes_sbp_main_and_age_interaction}A. Figure~\ref{fig:nhanes_sbp_main_and_age_interaction}B shows the SHAP interaction value of systolic blood pressure and age. As suggested by the coloring in Figure~\ref{fig:nhanes_sbp}, this interaction accounts for most of the vertical variance in the systolic blood pressure SHAP values.
%interaction value of age and sex on the y-axis, revealing an age dependence of the gender risk gap. Men's increased risk of death vs. women peaks at age 60 in the NHANES I dataset.

\begin{figure}
  \centering
  \includegraphics[width=0.45\textwidth]{figures/nhanes_sbp_main_and_age_interaction.pdf}
  \caption{SHAP interaction values separate the impact of systolic blood pressure into main effects (A; Equation~\ref{eq:shapley_interaction_main}) and interaction effects (B; Equation~\ref{eq:shapley_interactions}). Systolic blood pressure has a strong interaction effect with age, so the sum of (A) and (B) nearly equals Figure \ref{fig:nhanes_sbp}. There is very little vertical dispersion in (A) since all the interaction effects have been removed.}
  \label{fig:nhanes_sbp_main_and_age_interaction}
\end{figure}

% \begin{figure}As in Figure~\ref{fig:nhanes_sbp}, each dot is a person, but these figures use 
%   \centering
%   \includegraphics[width=0.45\textwidth]{figures/nhanes_sbp_age_interaction.pdf}
%   \caption{As in Figures~\ref{fig:nhanes_bmi}~and~\ref{fig:nhanes_sbp} each dot is a person, but here the y-axis is a SHAP interaction value between age and sex. The plot shows how men's increased risk of dying vs. women changes across age. Note that this is just the interaction effect with age, there is still a large main effect that causes women to be at a lower risk no matter their age.}
%   \label{fig:nhanes_age_sex_interaction}
% \end{figure}


\section{Conclusion}
Several common feature attribution methods for tree ensembles are inconsistent, meaning they can lower a feature's assigned importance when the true impact of that feature actually increases. This can prevent the meaningful comparison of feature attribution values. In contrast, SHAP values consistently attribute feature importance, better align with human intuition, and better recover influential features. By presenting the first polynomial time algorithm for SHAP values in tree ensembles, we make them a practical replacement for previous methods. We further defined SHAP interaction values as a consistent way of measuring potentially hidden pairwise interaction relationships. Tree SHAP's exponential speed improvements open up new practical opportunities, such as supervised clustering, SHAP summary plots, and SHAP dependence plots, that advance our understanding of tree models.

\vspace{3.2pt}
\noindent {\it Acknowledgements:} Vadim Khotilovich for helpful feedback. 

\bibliographystyle{ACM-Reference-Format}
\bibliography{main}

\end{document}



\documentclass[sigconf]{acmart}

\usepackage{booktabs} % For formal tables


% Copyright
%\setcopyright{none}
%\setcopyright{acmcopyright}
%\setcopyright{acmlicensed}
\setcopyright{rightsretained}
%\setcopyright{usgov}
%\setcopyright{usgovmixed}
%\setcopyright{cagov}
%\setcopyright{cagovmixed}


% DOI
\acmDOI{10.475/123_4}

% ISBN
\acmISBN{123-4567-24-567/08/06}

%Conference
\acmConference[WOODSTOCK'97]{ACM Woodstock conference}{July 1997}{El
  Paso, Texas USA}
\acmYear{1997}
\copyrightyear{2016}


\acmArticle{4}
\acmPrice{15.00}

% These commands are optional
%\acmBooktitle{Transactions of the ACM Woodstock conference}
\editor{Jennifer B. Sartor}
\editor{Theo D'Hondt}
\editor{Wolfgang De Meuter}


\begin{document}
\title{SIG Proceedings Paper in LaTeX Format}
\titlenote{Produces the permission block, and
  copyright information}
\subtitle{Extended Abstract}
\subtitlenote{The full version of the author's guide is available as
  \texttt{acmart.pdf} document}


\author{Ben Trovato}
\authornote{Dr.~Trovato insisted his name be first.}
\orcid{1234-5678-9012}
\affiliation{%
  \institution{Institute for Clarity in Documentation}
  \streetaddress{P.O. Box 1212}
  \city{Dublin}
  \state{Ohio}
  \postcode{43017-6221}
}
\email{trovato@corporation.com}

\author{G.K.M. Tobin}
\authornote{The secretary disavows any knowledge of this author's actions.}
\affiliation{%
  \institution{Institute for Clarity in Documentation}
  \streetaddress{P.O. Box 1212}
  \city{Dublin}
  \state{Ohio}
  \postcode{43017-6221}
}
\email{webmaster@marysville-ohio.com}

\author{Lars Th{\o}rv{\"a}ld}
\authornote{This author is the
  one who did all the really hard work.}
\affiliation{%
  \institution{The Th{\o}rv{\"a}ld Group}
  \streetaddress{1 Th{\o}rv{\"a}ld Circle}
  \city{Hekla}
  \country{Iceland}}
\email{larst@affiliation.org}

\author{Valerie B\'eranger}
\affiliation{%
  \institution{Inria Paris-Rocquencourt}
  \city{Rocquencourt}
  \country{France}
}
\author{Aparna Patel}
\affiliation{%
 \institution{Rajiv Gandhi University}
 \streetaddress{Rono-Hills}
 \city{Doimukh}
 \state{Arunachal Pradesh}
 \country{India}}
\author{Huifen Chan}
\affiliation{%
  \institution{Tsinghua University}
  \streetaddress{30 Shuangqing Rd}
  \city{Haidian Qu}
  \state{Beijing Shi}
  \country{China}
}

\author{Charles Palmer}
\affiliation{%
  \institution{Palmer Research Laboratories}
  \streetaddress{8600 Datapoint Drive}
  \city{San Antonio}
  \state{Texas}
  \postcode{78229}}
\email{cpalmer@prl.com}

\author{John Smith}
\affiliation{\institution{The Th{\o}rv{\"a}ld Group}}
\email{jsmith@affiliation.org}

\author{Julius P.~Kumquat}
\affiliation{\institution{The Kumquat Consortium}}
\email{jpkumquat@consortium.net}

% The default list of authors is too long for headers.
\renewcommand{\shortauthors}{B. Trovato et al.}


\begin{abstract}
This paper provides a sample of a \LaTeX\ document which conforms,
somewhat loosely, to the formatting guidelines for
ACM SIG Proceedings.\footnote{This is an abstract footnote}
\end{abstract}

%
% The code below should be generated by the tool at
% http://dl.acm.org/ccs.cfm
% Please copy and paste the code instead of the example below.
%
\begin{CCSXML}
<ccs2012>
 <concept>
  <concept_id>10010520.10010553.10010562</concept_id>
  <concept_desc>Computer systems organization~Embedded systems</concept_desc>
  <concept_significance>500</concept_significance>
 </concept>
 <concept>
  <concept_id>10010520.10010575.10010755</concept_id>
  <concept_desc>Computer systems organization~Redundancy</concept_desc>
  <concept_significance>300</concept_significance>
 </concept>
 <concept>
  <concept_id>10010520.10010553.10010554</concept_id>
  <concept_desc>Computer systems organization~Robotics</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
 <concept>
  <concept_id>10003033.10003083.10003095</concept_id>
  <concept_desc>Networks~Network reliability</concept_desc>
  <concept_significance>100</concept_significance>
 </concept>
</ccs2012>
\end{CCSXML}

\ccsdesc[500]{Computer systems organization~Embedded systems}
\ccsdesc[300]{Computer systems organization~Redundancy}
\ccsdesc{Computer systems organization~Robotics}
\ccsdesc[100]{Networks~Network reliability}


\keywords{ACM proceedings, \LaTeX, text tagging}


\maketitle

\input{samplebody-conf}

\bibliographystyle{ACM-Reference-Format}
\bibliography{sample-bibliography}

\end{document}

\section{Introduction}

The \textit{proceedings} are the records of a conference.\footnote{This
  is a footnote}  ACM seeks
to give these conference by-products a uniform, high-quality
appearance.  To do this, ACM has some rigid requirements for the
format of the proceedings documents: there is a specified format
(balanced double columns), a specified set of fonts (Arial or
Helvetica and Times Roman) in certain specified sizes, a specified
live area, centered on the page, specified size of margins, specified
column width and gutter size.

\section{The Body of The Paper}
Typically, the body of a paper is organized into a hierarchical
structure, with numbered or unnumbered headings for sections,
subsections, sub-subsections, and even smaller sections.  The command
\texttt{{\char'134}section} that precedes this paragraph is part of
such a hierarchy.\footnote{This is a footnote.} \LaTeX\ handles the
numbering and placement of these headings for you, when you use the
appropriate heading commands around the titles of the headings.  If
you want a sub-subsection or smaller part to be unnumbered in your
output, simply append an asterisk to the command name.  Examples of
both numbered and unnumbered headings will appear throughout the
balance of this sample document.

Because the entire article is contained in the \textbf{document}
environment, you can indicate the start of a new paragraph with a
blank line in your input file; that is why this sentence forms a
separate paragraph.

\subsection{Type Changes and {\itshape Special} Characters}

We have already seen several typeface changes in this sample.  You can
indicate italicized words or phrases in your text with the command
\texttt{{\char'134}textit}; emboldening with the command
\texttt{{\char'134}textbf} and typewriter-style (for instance, for
computer code) with \texttt{{\char'134}texttt}.  But remember, you do
not have to indicate typestyle changes when such changes are part of
the \textit{structural} elements of your article; for instance, the
heading of this subsection will be in a sans serif\footnote{Another
  footnote here.  Let's make this a rather long one to see how it
  looks.} typeface, but that is handled by the document class file.
Take care with the use of\footnote{Another footnote.}  the
curly braces in typeface changes; they mark the beginning and end of
the text that is to be in the different typeface.

You can use whatever symbols, accented characters, or non-English
characters you need anywhere in your document; you can find a complete
list of what is available in the \textit{\LaTeX\ User's Guide}
\cite{Lamport:LaTeX}.

\subsection{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of
the three are discussed in the next sections.

\subsubsection{Inline (In-text) Equations}
A formula that appears in the running text is called an
inline or in-text formula.  It is produced by the
\textbf{math} environment, which can be
invoked with the usual \texttt{{\char'134}begin\,\ldots{\char'134}end}
construction or with the short form \texttt{\$\,\ldots\$}. You
can use any of the symbols and structures,
from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a
few examples of in-text equations in context. Notice how
this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsubsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols
and structures available in \LaTeX\@; this section will just
give a couple of examples of display equations in context.
First, consider the equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\subsection{Citations}
Citations to articles~\cite{bowman:reasoning,
clark:pct, braams:babel, herlihy:methodology},
conference proceedings~\cite{clark:pct} or maybe
books \cite{Lamport:LaTeX, salas:calculus} listed
in the Bibliography section of your
article will occur throughout the text of your article.
You should use BibTeX to automatically produce this bibliography;
you simply need to insert one of several citation commands with
a key of the item cited in the proper location in
the \texttt{.tex} file~\cite{Lamport:LaTeX}.
The key is a short reference you invent to uniquely
identify each work; in this sample document, the key is
the first author's surname and a
word from the title.  This identifying key is included
with each item in the \texttt{.bib} file for your article.

The details of the construction of the \texttt{.bib} file
are beyond the scope of this sample document, but more
information can be found in the \textit{Author's Guide},
and exhaustive details in the \textit{\LaTeX\ User's
Guide} by Lamport~\shortcite{Lamport:LaTeX}.

This article shows only the plainest form
of the citation command, using \texttt{{\char'134}cite}.

Some examples.  A paginated journal article \cite{Abril07}, an enumerated
journal article \cite{Cohen07}, a reference to an entire issue \cite{JCohen96},
a monograph (whole book) \cite{Kosiur01}, a monograph/whole book in a series (see 2a in spec. document)
\cite{Harel79}, a divisible-book such as an anthology or compilation \cite{Editor00}
followed by the same example, however we only output the series if the volume number is given
\cite{Editor00a} (so Editor00a's series should NOT be present since it has no vol. no.),
a chapter in a divisible book \cite{Spector90}, a chapter in a divisible book
in a series \cite{Douglass98}, a multi-volume work as book \cite{Knuth97},
an article in a proceedings (of a conference, symposium, workshop for example)
(paginated proceedings article) \cite{Andler79}, a proceedings article
with all possible elements \cite{Smith10}, an example of an enumerated
proceedings article \cite{VanGundy07},
an informally published work \cite{Harel78}, a doctoral dissertation \cite{Clarkson85},
a master's thesis: \cite{anisi03}, an online document / world wide web
resource \cite{Thornburg01, Ablamowicz07, Poker06}, a video game (Case 1) \cite{Obama08} and (Case 2) \cite{Novak03}
and \cite{Lee05} and (Case 3) a patent \cite{JoeScientist001},
work accepted for publication \cite{rous08}, 'YYYYb'-test for prolific author
\cite{SaeediMEJ10} and \cite{SaeediJETC10}. Other cites might contain
'duplicate' DOI and URLs (some SIAM articles) \cite{Kirschmer:2010:AEI:1958016.1958018}.
Boris / Barbara Beeton: multi-volume works as books
\cite{MR781536} and \cite{MR781537}.

A couple of citations with DOIs: \cite{2004:ITE:1009386.1010128,
  Kirschmer:2010:AEI:1958016.1958018}.

Online citations: \cite{TUGInstmem, Thornburg01, CTANacmart}.


\subsection{Tables}
Because tables cannot be split across pages, the best
placement for them is typically the top of the page
nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and
the table caption.  The contents of the table itself must go
in the \textbf{tabular} environment, to
be aligned properly in rows and columns, with the desired
horizontal and vertical rules.  Again, detailed instructions
on \textbf{tabular} material
are found in the \textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed
output of this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more desirable.
Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.


\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}
% end the environment with {table*}, NOTE not {table}!

It is strongly recommended to use the package booktabs~\cite{Fear05}
and follow its main principles of typography with respect to tables:
\begin{enumerate}
\item Never, ever use vertical rules.
\item Never use double rules.
\end{enumerate}
It is also a good idea not to overuse horizontal rules.


\subsection{Figures}

Like tables, figures cannot be split across pages; the best placement
for them is typically the top or the bottom of the page nearest their
initial cite.  To ensure this proper ``floating'' placement of
figures, use the environment \textbf{figure} to enclose the figure and
its caption.

This sample document contains examples of \texttt{.eps} files to be
displayable with \LaTeX.  If you work with pdf\LaTeX, use files in the
\texttt{.pdf} format.  Note that most modern \TeX\ systems will convert
\texttt{.eps} to \texttt{.pdf} for you on the fly.  More details on
each of these are found in the \textit{Author's Guide}.

\begin{figure}
\includegraphics{fly}
\caption{A sample black and white graphic.}
\end{figure}

\begin{figure}
\includegraphics[height=1in, width=1in]{fly}
\caption{A sample black and white graphic
that has been resized with the \texttt{includegraphics} command.}
\end{figure}


As was the case with tables, you may want a figure that spans two
columns.  To do this, and still to ensure proper ``floating''
placement of tables, use the environment \textbf{figure*} to enclose
the figure and its caption.  And don't forget to end the environment
with \textbf{figure*}, not \textbf{figure}!

\begin{figure*}
\includegraphics{flies}
\caption{A sample black and white graphic
that needs to span two columns of text.}
\end{figure*}


\begin{figure}
\includegraphics[height=1in, width=1in]{rosette}
\caption{A sample black and white graphic that has
been resized with the \texttt{includegraphics} command.}
\end{figure}

\subsection{Theorem-like Constructs}

Other common constructs that may occur in your article are the forms
for logical constructs like theorems, axioms, corollaries and proofs.
ACM uses two types of these constructs:  theorem-like and
definition-like.

Here is a theorem:
\begin{theorem}
  Let $f$ be continuous on $[a,b]$.  If $G$ is
  an antiderivative for $f$ on $[a,b]$, then
  \begin{displaymath}
    \int^b_af(t)\,dt = G(b) - G(a).
  \end{displaymath}
\end{theorem}

Here is a definition:
\begin{definition}
  If $z$ is irrational, then by $e^z$ we mean the
  unique number that has
  logarithm $z$:
  \begin{displaymath}
    \log e^z = z.
  \end{displaymath}
\end{definition}

The pre-defined theorem-like constructs are \textbf{theorem},
\textbf{conjecture}, \textbf{proposition}, \textbf{lemma} and
\textbf{corollary}.  The pre-defined de\-fi\-ni\-ti\-on-like constructs are
\textbf{example} and \textbf{definition}.  You can add your own
constructs using the \textsl{amsthm} interface~\cite{Amsthm15}.  The
styles used in the \verb|\theoremstyle| command are \textbf{acmplain}
and \textbf{acmdefinition}.

Another construct is \textbf{proof}, for example,

\begin{proof}
  Suppose on the contrary there exists a real number $L$ such that
  \begin{displaymath}
    \lim_{x\rightarrow\infty} \frac{f(x)}{g(x)} = L.
  \end{displaymath}
  Then
  \begin{displaymath}
    l=\lim_{x\rightarrow c} f(x)
    = \lim_{x\rightarrow c}
    \left[ g{x} \cdot \frac{f(x)}{g(x)} \right ]
    = \lim_{x\rightarrow c} g(x) \cdot \lim_{x\rightarrow c}
    \frac{f(x)}{g(x)} = 0\cdot L = 0,
  \end{displaymath}
  which contradicts our assumption that $l\neq 0$.
\end{proof}

\section{Conclusions}
This paragraph will end the body of this sample document.
Remember that you might still have Acknowledgments or
Appendices; brief samples of these
follow.  There is still the Bibliography to deal with; and
we will make a disclaimer about that here: with the exception
of the reference to the \LaTeX\ book, the citations in
this paper are to articles which have nothing to
do with the present subject and are used as
examples only.
%\end{document}  % This is where a 'short' article might terminate



\appendix
%Appendix A
\section{Headings in Appendices}
The rules about hierarchical headings discussed above for
the body of the article are different in the appendices.
In the \textbf{appendix} environment, the command
\textbf{section} is used to
indicate the start of each Appendix, with alphabetic order
designation (i.e., the first is A, the second B, etc.) and
a title (if you include one).  So, if you need
hierarchical structure
\textit{within} an Appendix, start with \textbf{subsection} as the
highest level. Here is an outline of the body of this
document in Appendix-appropriate form:
\subsection{Introduction}
\subsection{The Body of the Paper}
\subsubsection{Type Changes and  Special Characters}
\subsubsection{Math Equations}
\paragraph{Inline (In-text) Equations}
\paragraph{Display Equations}
\subsubsection{Citations}
\subsubsection{Tables}
\subsubsection{Figures}
\subsubsection{Theorem-like Constructs}
\subsubsection*{A Caveat for the \TeX\ Expert}
\subsection{Conclusions}
\subsection{References}
Generated by bibtex from your \texttt{.bib} file.  Run latex,
then bibtex, then latex twice (to resolve references)
to create the \texttt{.bbl} file.  Insert that \texttt{.bbl}
file into the \texttt{.tex} source file and comment out
the command \texttt{{\char'134}thebibliography}.
% This next section command marks the start of
% Appendix B, and does not continue the present hierarchy
\section{More Help for the Hardy}

Of course, reading the source code is always useful.  The file
\path{acmart.pdf} contains both the user guide and the commented
code.

\begin{acks}
  The authors would like to thank Dr. Yuhua Li for providing the
  MATLAB code of the \textit{BEPS} method.

  The authors would also like to thank the anonymous referees for
  their valuable comments and helpful suggestions. The work is
  supported by the \grantsponsor{GS501100001809}{National Natural
    Science Foundation of
    China}{http://dx.doi.org/10.13039/501100001809} under Grant
  No.:~\grantnum{GS501100001809}{61273304}
  and~\grantnum[http://www.nnsf.cn/youngscientists]{GS501100001809}{Young
    Scientists' Support Program}.

\end{acks}

%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].

\begin{thebibliography}{00}

%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command.  Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}.  The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip}   % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip}           % plain TeX syntax
%%%
%%% ====================================================================

\ifx \showCODEN    \undefined \def \showCODEN     #1{\unskip}     \fi
\ifx \showDOI      \undefined \def \showDOI       #1{#1}\fi
\ifx \showISBNx    \undefined \def \showISBNx     #1{\unskip}     \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii  #1{\unskip}     \fi
\ifx \showISSN     \undefined \def \showISSN      #1{\unskip}     \fi
\ifx \showLCCN     \undefined \def \showLCCN      #1{\unskip}     \fi
\ifx \shownote     \undefined \def \shownote      #1{#1}          \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1}   \fi
\ifx \showURL      \undefined \def \showURL       {\relax}        \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}

\bibitem[\protect\citeauthoryear{Auret and Aldrich}{Auret and Aldrich}{2011}]%
        {auret2011empirical}
\bibfield{author}{\bibinfo{person}{Lidia Auret} {and} \bibinfo{person}{Chris
  Aldrich}.} \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Empirical comparison of tree ensemble variable
  importance measures}.
\newblock \bibinfo{journal}{{\em Chemometrics and Intelligent Laboratory
  Systems\/}} \bibinfo{volume}{105}, \bibinfo{number}{2}
  (\bibinfo{year}{2011}), \bibinfo{pages}{157--170}.
\newblock


\bibitem[\protect\citeauthoryear{Baehrens, Schroeter, Harmeling, Kawanabe,
  Hansen, and M{\~A}{\v{z}}ller}{Baehrens et~al\mbox{.}}{2010}]%
        {baehrens2010explain}
\bibfield{author}{\bibinfo{person}{David Baehrens}, \bibinfo{person}{Timon
  Schroeter}, \bibinfo{person}{Stefan Harmeling}, \bibinfo{person}{Motoaki
  Kawanabe}, \bibinfo{person}{Katja Hansen}, {and}
  \bibinfo{person}{Klaus-Robert M{\~A}{\v{z}}ller}.}
  \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{How to explain individual classification
  decisions}.
\newblock \bibinfo{journal}{{\em Journal of Machine Learning Research\/}}
  \bibinfo{volume}{11}, \bibinfo{number}{Jun} (\bibinfo{year}{2010}),
  \bibinfo{pages}{1803--1831}.
\newblock


\bibitem[\protect\citeauthoryear{Breiman, Friedman, Stone, and Olshen}{Breiman
  et~al\mbox{.}}{1984}]%
        {breiman1984classification}
\bibfield{author}{\bibinfo{person}{Leo Breiman}, \bibinfo{person}{Jerome
  Friedman}, \bibinfo{person}{Charles~J Stone}, {and}
  \bibinfo{person}{Richard~A Olshen}.} \bibinfo{year}{1984}\natexlab{}.
\newblock \bibinfo{booktitle}{{\em Classification and regression trees}}.
\newblock \bibinfo{publisher}{CRC press}.
\newblock


\bibitem[\protect\citeauthoryear{Celik, Logsdon, and Lee}{Celik
  et~al\mbox{.}}{2014}]%
        {celik2014efficient}
\bibfield{author}{\bibinfo{person}{Safiye Celik}, \bibinfo{person}{Benjamin
  Logsdon}, {and} \bibinfo{person}{Su-In Lee}.}
  \bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Efficient dimensionality reduction for
  high-dimensional network estimation}. In \bibinfo{booktitle}{{\em
  International Conference on Machine Learning}}. \bibinfo{pages}{1953--1961}.
\newblock


\bibitem[\protect\citeauthoryear{Chebrolu, Abraham, and Thomas}{Chebrolu
  et~al\mbox{.}}{2005}]%
        {chebrolu2005feature}
\bibfield{author}{\bibinfo{person}{S Chebrolu}, \bibinfo{person}{A Abraham},
  {and} \bibinfo{person}{J Thomas}.} \bibinfo{year}{2005}\natexlab{}.
\newblock \showarticletitle{Feature deduction and ensemble design of intrusion
  detection systems}.
\newblock \bibinfo{journal}{{\em Computers \& security\/}}
  \bibinfo{volume}{24}, \bibinfo{number}{4} (\bibinfo{year}{2005}),
  \bibinfo{pages}{295--307}.
\newblock


\bibitem[\protect\citeauthoryear{Chen and Guestrin}{Chen and Guestrin}{2016}]%
        {chen2016xgboost}
\bibfield{author}{\bibinfo{person}{Tianqi Chen} {and} \bibinfo{person}{Carlos
  Guestrin}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{XGBoost: A scalable tree boosting system}. In
  \bibinfo{booktitle}{{\em Proceedings of the 22Nd ACM SIGKDD International
  Conference on Knowledge Discovery and Data Mining}}. ACM,
  \bibinfo{pages}{785--794}.
\newblock


\bibitem[\protect\citeauthoryear{Cox, Feldman, Golden, Lane, Madans, Mussolino,
  and Rothwell}{Cox et~al\mbox{.}}{1997}]%
        {cox1997plan}
\bibfield{author}{\bibinfo{person}{Christine~S Cox}, \bibinfo{person}{Jacob~J
  Feldman}, \bibinfo{person}{Cordell~D Golden}, \bibinfo{person}{Madelyn~A
  Lane}, \bibinfo{person}{Jennifer~H Madans}, \bibinfo{person}{Michael~E
  Mussolino}, {and} \bibinfo{person}{Sandra~T Rothwell}.}
  \bibinfo{year}{1997}\natexlab{}.
\newblock \showarticletitle{Plan and operation of the NHANES I Epidemiologic
  Followup Study, 1992}.
\newblock  (\bibinfo{year}{1997}).
\newblock


\bibitem[\protect\citeauthoryear{Crowdflower}{Crowdflower}{2015}]%
        {airline_tweets}
\bibfield{author}{\bibinfo{person}{Crowdflower}.}
  \bibinfo{year}{2015}\natexlab{}.
\newblock \bibinfo{title}{Twitter US Airline Sentiment}.
\newblock
  \bibinfo{howpublished}{\url{https://www.kaggle.com/crowdflower/twitter-airline-sentiment}}.
    (\bibinfo{year}{2015}).
\newblock
\newblock
\shownote{Accessed: 2018-02-06.}


\bibitem[\protect\citeauthoryear{Datta, Sen, and Zick}{Datta
  et~al\mbox{.}}{2016}]%
        {datta2016algorithmic}
\bibfield{author}{\bibinfo{person}{Anupam Datta}, \bibinfo{person}{Shayak Sen},
  {and} \bibinfo{person}{Yair Zick}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Algorithmic transparency via quantitative input
  influence: Theory and experiments with learning systems}. In
  \bibinfo{booktitle}{{\em Security and Privacy (SP), 2016 IEEE Symposium on}}.
  IEEE, \bibinfo{pages}{598--617}.
\newblock


\bibitem[\protect\citeauthoryear{D{\'\i}az-Uriarte and
  De~Andres}{D{\'\i}az-Uriarte and De~Andres}{2006}]%
        {diaz2006gene}
\bibfield{author}{\bibinfo{person}{R D{\'\i}az-Uriarte} {and}
  \bibinfo{person}{S De~Andres}.} \bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Gene selection and classification of microarray
  data using random forest}.
\newblock \bibinfo{journal}{{\em BMC bioinformatics\/}} \bibinfo{volume}{7},
  \bibinfo{number}{1} (\bibinfo{year}{2006}), \bibinfo{pages}{3}.
\newblock


\bibitem[\protect\citeauthoryear{Friedman, Hastie, and Tibshirani}{Friedman
  et~al\mbox{.}}{2001}]%
        {friedman2001elements}
\bibfield{author}{\bibinfo{person}{Jerome Friedman}, \bibinfo{person}{Trevor
  Hastie}, {and} \bibinfo{person}{Robert Tibshirani}.}
  \bibinfo{year}{2001}\natexlab{}.
\newblock \bibinfo{booktitle}{{\em The elements of statistical learning}}.
  Vol.~\bibinfo{volume}{1}.
\newblock \bibinfo{publisher}{Springer series in statistics Springer, Berlin}.
\newblock


\bibitem[\protect\citeauthoryear{Fujimoto, Kojadinovic, and Marichal}{Fujimoto
  et~al\mbox{.}}{2006}]%
        {fujimoto2006axiomatic}
\bibfield{author}{\bibinfo{person}{Katsushige Fujimoto}, \bibinfo{person}{Ivan
  Kojadinovic}, {and} \bibinfo{person}{Jean-Luc Marichal}.}
  \bibinfo{year}{2006}\natexlab{}.
\newblock \showarticletitle{Axiomatic characterizations of probabilistic and
  cardinal-probabilistic interaction indices}.
\newblock \bibinfo{journal}{{\em Games and Economic Behavior\/}}
  \bibinfo{volume}{55}, \bibinfo{number}{1} (\bibinfo{year}{2006}),
  \bibinfo{pages}{72--99}.
\newblock


\bibitem[\protect\citeauthoryear{Irrthum, Wehenkel, Geurts,
  et~al\mbox{.}}{Irrthum et~al\mbox{.}}{2010}]%
        {irrthum2010inferring}
\bibfield{author}{\bibinfo{person}{A Irrthum}, \bibinfo{person}{L Wehenkel},
  \bibinfo{person}{P Geurts}, {et~al\mbox{.}}} \bibinfo{year}{2010}\natexlab{}.
\newblock \showarticletitle{Inferring regulatory networks from expression data
  using tree-based methods}.
\newblock \bibinfo{journal}{{\em PloS one\/}} \bibinfo{volume}{5},
  \bibinfo{number}{9} (\bibinfo{year}{2010}), \bibinfo{pages}{e12776}.
\newblock


\bibitem[\protect\citeauthoryear{Ishwaran et~al\mbox{.}}{Ishwaran
  et~al\mbox{.}}{2007}]%
        {ishwaran2007variable}
\bibfield{author}{\bibinfo{person}{Hemant Ishwaran} {et~al\mbox{.}}}
  \bibinfo{year}{2007}\natexlab{}.
\newblock \showarticletitle{Variable importance in binary regression trees and
  forests}.
\newblock \bibinfo{journal}{{\em Electronic Journal of Statistics\/}}
  \bibinfo{volume}{1} (\bibinfo{year}{2007}), \bibinfo{pages}{519--537}.
\newblock


\bibitem[\protect\citeauthoryear{Lichman}{Lichman}{2013}]%
        {Lichman2013}
\bibfield{author}{\bibinfo{person}{M. Lichman}.}
  \bibinfo{year}{2013}\natexlab{}.
\newblock \bibinfo{title}{{UCI} ML Repository}.
\newblock   (\bibinfo{year}{2013}).
\newblock
\showURL{%
\url{http://archive.ics.uci.edu/ml}}


\bibitem[\protect\citeauthoryear{Lundberg and Lee}{Lundberg and Lee}{2017}]%
        {lundberg2017unified}
\bibfield{author}{\bibinfo{person}{Scott~M Lundberg} {and}
  \bibinfo{person}{Su-In Lee}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{A Unified Approach to Interpreting Model
  Predictions}.
\newblock In \bibinfo{booktitle}{{\em Advances in Neural Information Processing
  Systems 30}}. \bibinfo{publisher}{Curran Associates, Inc.},
  \bibinfo{pages}{4768--4777}.
\newblock
\showURL{%
\url{http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf}}


\bibitem[\protect\citeauthoryear{Lundberg, Nair, Vavilala, Horibe, Eisses,
  Adams, Liston, Low, Newman, Kim, et~al\mbox{.}}{Lundberg
  et~al\mbox{.}}{2017}]%
        {lundberg2017explainable}
\bibfield{author}{\bibinfo{person}{Scott~M Lundberg}, \bibinfo{person}{Bala
  Nair}, \bibinfo{person}{Monica~S Vavilala}, \bibinfo{person}{Mayumi Horibe},
  \bibinfo{person}{Michael~J Eisses}, \bibinfo{person}{Trevor Adams},
  \bibinfo{person}{David~E Liston}, \bibinfo{person}{Daniel King-Wai Low},
  \bibinfo{person}{Shu-Fang Newman}, \bibinfo{person}{Jerry Kim},
  {et~al\mbox{.}}} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Explainable machine learning predictions to help
  anesthesiologists prevent hypoxemia during surgery}.
\newblock \bibinfo{journal}{{\em bioRxiv\/}} (\bibinfo{year}{2017}),
  \bibinfo{pages}{206540}.
\newblock


\bibitem[\protect\citeauthoryear{Miller}{Miller}{1973}]%
        {miller1973plan}
\bibfield{author}{\bibinfo{person}{Henry~W Miller}.}
  \bibinfo{year}{1973}\natexlab{}.
\newblock \showarticletitle{Plan and operation of the health and nutrition
  examination survey, United States, 1971-1973}.
\newblock \bibinfo{journal}{{\em DHEW publication no.(PHS)-Dept. of Health,
  Education, and Welfare (USA)\/}} (\bibinfo{year}{1973}).
\newblock


\bibitem[\protect\citeauthoryear{Mirra, Heyman, McKeel, Sumi, Crain, Brownlee,
  Vogel, Hughes, Van~Belle, Berg, et~al\mbox{.}}{Mirra et~al\mbox{.}}{1991}]%
        {mirra1991consortium}
\bibfield{author}{\bibinfo{person}{Suzanne~S Mirra}, \bibinfo{person}{A
  Heyman}, \bibinfo{person}{D McKeel}, \bibinfo{person}{SM Sumi},
  \bibinfo{person}{Barbara~J Crain}, \bibinfo{person}{LM Brownlee},
  \bibinfo{person}{FS Vogel}, \bibinfo{person}{JP Hughes}, \bibinfo{person}{G
  Van~Belle}, \bibinfo{person}{L Berg}, {et~al\mbox{.}}}
  \bibinfo{year}{1991}\natexlab{}.
\newblock \showarticletitle{The Consortium to Establish a Registry for
  Alzheimer's Disease (CERAD) Part II. Stand. of the neuropathologic
  assessmen== of Alzheimer's disease}.
\newblock \bibinfo{journal}{{\em Neurology\/}} \bibinfo{volume}{41},
  \bibinfo{number}{4} (\bibinfo{year}{1991}), \bibinfo{pages}{479--479}.
\newblock


\bibitem[\protect\citeauthoryear{Pedregosa, Varoquaux, Gramfort, Michel,
  Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg,
  et~al\mbox{.}}{Pedregosa et~al\mbox{.}}{2011}]%
        {pedregosa2011scikit}
\bibfield{author}{\bibinfo{person}{F Pedregosa}, \bibinfo{person}{G Varoquaux},
  \bibinfo{person}{A Gramfort}, \bibinfo{person}{V Michel}, \bibinfo{person}{B
  Thirion}, \bibinfo{person}{O Grisel}, \bibinfo{person}{M Blondel},
  \bibinfo{person}{P Prettenhofer}, \bibinfo{person}{R Weiss},
  \bibinfo{person}{V Dubourg}, {et~al\mbox{.}}}
  \bibinfo{year}{2011}\natexlab{}.
\newblock \showarticletitle{Scikit-learn: Machine learning in Python}.
\newblock \bibinfo{journal}{{\em JMLR\/}} \bibinfo{volume}{12},
  \bibinfo{number}{Oct} (\bibinfo{year}{2011}), \bibinfo{pages}{2825--2830}.
\newblock


\bibitem[\protect\citeauthoryear{Ribeiro, Singh, and Guestrin}{Ribeiro
  et~al\mbox{.}}{2016}]%
        {ribeiro2016should}
\bibfield{author}{\bibinfo{person}{Marco~Tulio Ribeiro},
  \bibinfo{person}{Sameer Singh}, {and} \bibinfo{person}{Carlos Guestrin}.}
  \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Why should i trust you?: Explaining the predictions
  of any classifier}. In \bibinfo{booktitle}{{\em Proceedings of the 22nd ACM
  SIGKDD}}. ACM, \bibinfo{pages}{1135--1144}.
\newblock


\bibitem[\protect\citeauthoryear{Ridgeway}{Ridgeway}{2010}]%
        {ridgeway2010generalized}
\bibfield{author}{\bibinfo{person}{Greg Ridgeway}.}
  \bibinfo{year}{2010}\natexlab{}.
\newblock \bibinfo{title}{Generalized boosted regression models. Documentation
  on the R Package ‘gbm’, version 1.6--3}.
\newblock   (\bibinfo{year}{2010}).
\newblock


\bibitem[\protect\citeauthoryear{Rodenburg, Heidema, Boer, Bovee-Oudenhoven,
  Feskens, Mariman, and Keijer}{Rodenburg et~al\mbox{.}}{2008}]%
        {rodenburg2008framework}
\bibfield{author}{\bibinfo{person}{W Rodenburg}, \bibinfo{person}{G Heidema},
  \bibinfo{person}{J Boer}, \bibinfo{person}{I Bovee-Oudenhoven},
  \bibinfo{person}{E Feskens}, \bibinfo{person}{E Mariman}, {and}
  \bibinfo{person}{J Keijer}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{A framework to identify physiological responses in
  microarray-based gene expression studies: selection and interpretation of
  biologically relevant genes}.
\newblock \bibinfo{journal}{{\em Physiological genomics\/}}
  \bibinfo{volume}{33}, \bibinfo{number}{1} (\bibinfo{year}{2008}),
  \bibinfo{pages}{78--90}.
\newblock


\bibitem[\protect\citeauthoryear{Saabas}{Saabas}{2014}]%
        {path_blog}
\bibfield{author}{\bibinfo{person}{Ando Saabas}.}
  \bibinfo{year}{2014}\natexlab{}.
\newblock \bibinfo{title}{Interpreting random forests}.
\newblock
  \bibinfo{howpublished}{\url{http://blog.datadive.net/interpreting-random-forests/}}.
    (\bibinfo{year}{2014}).
\newblock
\newblock
\shownote{Accessed: 2017-06-15.}


\bibitem[\protect\citeauthoryear{Sandri and Zuccolotto}{Sandri and
  Zuccolotto}{2008}]%
        {sandri2008bias}
\bibfield{author}{\bibinfo{person}{Marco Sandri} {and} \bibinfo{person}{Paola
  Zuccolotto}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{A bias correction algorithm for the Gini variable
  importance measure in classification trees}.
\newblock \bibinfo{journal}{{\em Journal of Computational and Graphical
  Statistics\/}} \bibinfo{volume}{17}, \bibinfo{number}{3}
  (\bibinfo{year}{2008}), \bibinfo{pages}{611--628}.
\newblock


\bibitem[\protect\citeauthoryear{Strobl, Boulesteix, Kneib, Augustin, and
  Zeileis}{Strobl et~al\mbox{.}}{2008}]%
        {strobl2008conditional}
\bibfield{author}{\bibinfo{person}{C Strobl}, \bibinfo{person}{A Boulesteix},
  \bibinfo{person}{T Kneib}, \bibinfo{person}{T Augustin}, {and}
  \bibinfo{person}{A Zeileis}.} \bibinfo{year}{2008}\natexlab{}.
\newblock \showarticletitle{Conditional variable importance for random
  forests}.
\newblock \bibinfo{journal}{{\em BMC bioinformatics\/}} \bibinfo{volume}{9},
  \bibinfo{number}{1} (\bibinfo{year}{2008}), \bibinfo{pages}{307}.
\newblock


\bibitem[\protect\citeauthoryear{{\v{S}}trumbelj and Kononenko}{{\v{S}}trumbelj
  and Kononenko}{2014}]%
        {vstrumbelj2014explaining}
\bibfield{author}{\bibinfo{person}{Erik {\v{S}}trumbelj} {and}
  \bibinfo{person}{Igor Kononenko}.} \bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Explaining prediction models and individual
  predictions with feature contributions}.
\newblock \bibinfo{journal}{{\em Knowledge and information systems\/}}
  \bibinfo{volume}{41}, \bibinfo{number}{3} (\bibinfo{year}{2014}),
  \bibinfo{pages}{647--665}.
\newblock


\end{thebibliography}