Explaining the Predictions of Any Image Classifier via Decision Trees
Extracted source text · arXiv 1911.01058
% Template for ICASSP-2020 paper; to be used with:
% spconf.sty - ICASSP/ICIP LaTeX style file, and
% IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}
%\usepackage[ruled]{algorithm2e}
\usepackage{threeparttable}
\usepackage{dcolumn}
\usepackage{booktabs}
\usepackage{amsmath,graphicx}
\usepackage{url}
\usepackage{cite}
\usepackage{psfrag}
\usepackage{amsfonts,amssymb}
\usepackage{amsbsy}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{array}
\usepackage{multirow}
\usepackage{bm}
\usepackage{subfigure}
\usepackage{verbatim}
\usepackage{colortbl}
\usepackage{stfloats}
%\usepackage{algorithmicx}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{setspace}
%\usepackage{algorithm2e,setspace}
\usepackage{bigstrut}
\usepackage{amsmath}
\usepackage{mathtools}
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}
% Title.
% ------
\title{Explaining the Predictions of Any Image Classifier via Decision Trees}
%
% Single address.
% ---------------
%\name{Author(s) Name(s)\thanks{Thanks to XYZ agency for funding.}}
%\address{Author Affiliation(s)}
\name{Sheng Shi$^{\dagger}$ \qquad Xinfeng Zhang$^{\star}$ \qquad Haisheng Li$^{\dagger}$ \qquad Wei Fan$^{\dagger}$}
\address{$^{\dagger}$ AI Laboratory, Lenovo Research, Beijing 100094, China \\
$^{\star}$ University of Chinese Academy of Sciences, Beijing 100049, China}
%
% For example:
% ------------
%\address{School\\
% Department\\
% Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
% {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
% {School A-B\\
% Department A-B\\
% Address A-B}
% {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
% while at ...}}
% {School C-D\\
% Department C-D\\
% Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Despite outstanding contribution to the significant progress of Artificial Intelligence (AI), deep learning models remain mostly black boxes, which are extremely weak in explainability of the reasoning process and prediction results. Explainability is not only a gateway between AI and society but also a powerful tool to detect flaws in the model and biases in the data. Local Interpretable Model-agnostic Explanation (LIME) is a recent approach that uses a linear regression model to form a local explanation for the individual prediction result. However, being so restricted and usually over-simplifying the relationships, linear models fail in situations where nonlinear associations and interactions exist among features and prediction results. This paper proposes an extended Decision Tree-based LIME (TLIME) approach, which uses a decision tree model to form an interpretable representation that is locally faithful to the original model. The new approach can capture nonlinear interactions among features in the data and creates plausible explanations. Various experiments show that the TLIME explanation of multiple black-box models can achieve more reliable performance in terms of understandability, fidelity, and efficiency.
\end{abstract}
%
\begin{keywords}
Explainable AI, Interpretable Model, Decision Tree, Local Fidelity, Model Agnostic
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
In recent years, the fast-growing computing power, enormous consumer and commercial data, and emerging advanced machine learning algorithms jointly stimulate the prosperous of AI \cite{AI01}\cite{AI02}, which has gone from a science-fiction dream to a critical part of our daily life. Compared to traditional machine learning methods, deep learning has achieved superior performance in perception tasks such as object detection and classification. However, because of the nested non-linear structure, deep learning models usually remain black boxes that are particularly weak in the explainability of the reasoning process and prediction results. In many real-world mission-critical applications, transparency of deep learning models and explainability of the model outputs are essential and necessary in their real deployment process.
Explanable AI is not only a gateway between AI and society but also a powerful tool to detect flaws in the model and biases in the data. The development of techniques on explainability and transparency of deep learning models has recently received much attention in the research community \cite{One01}\cite{One02}\cite{One03}\cite{One04}. The relevant research roughly falls into two categories: global explainability and local explainability. Global explainability aims at making the reasoning process wholly transparent and comprehensive \cite{global03}\cite{global04}, while local explainability focuses on extracting input regions that are highly sensitive to the network output to provide explanations for each decision \cite{local01}\cite{local02}\cite{local03}\cite{local04}.
An effective way to achieve explainability is to use a light-weight function family to create interpretable models. Local interpretable model-agnostic explanations (LIME) identify an interpretable model over the human-interpretable representation that is locally faithful to the original model \cite{local01}. LIME adopts the linear regression as its interpretable function, which represents the prediction as a linear combination of a few selected features to make the prediction process transparent. However, being so restricted and usually over-simplifying the relationships, linear regression models fail in some situations where non-linear associations and interactions exist among features and prediction results %\cite{ref01}.
In this paper, we propose a Decision Tree-based Local Interpretable Model-agnostic Explanation (TLIME). The decision tree structure creates good explanations as the data ends up in distinct groups that are often easy to understand. Moreover, the tree structure can capture interactions between features in the data. We perform various experiments on explaining two black-box models, the random-forest classifier and Google's pre-trained Inception neural network\cite{Inception}. The results show that decision tree explanations achieve more reliable performance than original LIME in terms of understandability, fidelity, and efficiency.
% We organize the remainder of the paper as follows. Section II compares two representative interpretable models, the linear regression and the decision tree model, and Section III describes the proposed TLIME approach. We show experimental results of TLIME and LIME on a random-forest classifier and Google's pre-trained Inception neural network in Section IV and conclude the paper in Section V.
%section III describes the proposed local decision tree based interpretable model-agnostic explanation (TLIME) model.
%Local interpretable model-agnostic explanations (LIME) identify an interpretable model over the interpretable representation that is locally faithful to the classifier \cite{local01}. The interpretable model of LIME is sparse linear explanations.
%such as linear regression and decision tree.
%If you suspect feature interactions or a nonlinear association of a feature with the target value
%Using a subset of algorithms to create interpretable models is a common way to achieve interpretability. Linear regression and decision tree are commonly used to create interpretable models
%e easiest way to achieve interpretability is to use only a subset of algorithms
%The LIME model \cite{local01} is trained to
%of model-agnostic methods and model-specific methods.
%Advances on communication and transmission techniques over Internet have been witnessed under the background of the big data.
\section{Interpretable models}
\label{sec:format}
Using a subset of algorithms from a light-weight function family to create interpretable models is an effective way to achieve interpretability. In this section, we analyze two representative interpretable models - the linear regression model and the decision tree model. Table~\ref{tab:1} shows the properties of two interpretable models. The linear regression displays the prediction as a linear combination of features, while the decision tree represents the reasoning process in a hierarchical structure, which is suitable for capturing the nonlinear association between features and predictions. The monotonicity constraint shown in both models is necessary to ensure the consistency between a feature and the target outcome. Moreover, the decision tree model can automatically capture the diverse interactions between features to predict the target outcome, applicable to both classification and regression tasks. %In this paper, we mainly focus on the image classifier model.
%\begin{table}[htbp]
% \centering
% \caption{The properties of linear regression model and decision tree model}
%
% \begin{tabular}{|c|c|c|c|}
% %\begin{tabular}{|c|c|c|c|}
% \hline
% Models & Linear & Monotone & Interaction \bigstrut\\
% \hline
% Line regression & Yes & Yes & No
% \bigstrut\\
% \hline
% Decision tree & No & Some & Yes \bigstrut\\
% \hline
% \end{tabular}%
% \label{tab:1}%
%\end{table}
\begin{table*}
\centering\footnotesize
\caption{\small{The properties of linear regression model and decision tree model}}
\begin{tabular}{|c|c|c|c|c|}
%\begin{tabular}{|c|c|c|c|}
\hline
Models & Linearity & Monotonicity & Feature Interaction & Task \bigstrut\\
\hline
Line regression & Yes & Yes & No & Regression
\bigstrut\\
\hline
Decision trees & No & Some & Yes & Classification, Regression\bigstrut\\
\hline
\end{tabular}%
\label{tab:1}%
\end{table*}
Depending on the different criteria, various algorithms are capable of constructing a decision tree. The CART \cite{ref02} is the most popular algorithm which can handle both classification and regression tasks. In this paper, we mainly construct regression decision trees to explain the prediction probability of the image classifier. Figure~\ref{fig:1} illustrates a simple regression tree to explain image classification prediction made by Google's Inception neural network. The predicted top $1$ class label is $African$ $chanmeleon$ $(p=0.9935)$. The highlighted superpixels give intuition as to why the model would choose that class. The decision tree shows that if feature $28$, $22$, and $30$ exist, then the prediction probability is $0.991$, which is the mean value of the instances $y$ in this node. Moreover, The importance of the three features is $0.7164$, $0.0709$, and $0.0259$, showing the contribution of the three features in improving the variance.
%The instance needed to be explained is circled in red.
%For the regression tasks, CART takes a feature and determines which cut-off point minimizes the mean squared error. After determining the best cutoff of each feature, the algorithm selects the feature for splitting that would result in the best partition. Then, CART continues search-and-split recursively in new nodes until a stop criterion is reached.
%The relationship between the outcome $y$ of decision tree and features $x$ is described as following:
%\begin{equation}
%\hat(y)=argmin\quad{L(f,g)+ dep(g)}
%\end{equation}
%The decision tree can be traversed to come to a final decision.
%
%
% (1) introduce cart combined with the experiment figure. (2)Interpretation with feature importance and tree decomposition. (3)summary
%Decision tree model splits the data multiple times according to certain cutoff values in the features
%Decision tree can automatically include interactions between features to predict the target outcome.
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.9\textwidth]{figures/process/mnist_tree}}
%\end{minipage}
%\caption{decision tree}
%\label{fig:1}
%\end{figure}
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.9\textwidth]{figures/process/mnist_tree_2}}
%\end{minipage}
%\caption{decision tree}
%\label{fig:1}
%\end{figure}
\begin{figure}
\begin{minipage}{1\linewidth}
\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon_simple.pdf}}
\end{minipage}
\caption{\small{A simple regression decision tree explains the image classification results given by Google's Inception neural network. The top $1$ output class label is $African$ $chanmeleon$ with the prediction probability $(p=0.9935)$.}}
\vspace{-10pt}
\label{fig:1}
\end{figure}
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.8\textwidth]{figures/process/mnist_tree_1}}
%\end{minipage}
%\caption{decision tree}
%\label{fig:1}
%\end{figure}
\section{The TLIME Approach}
\label{sec:pagestyle}
\subsection{Characteristics of TLIME}
Despite the fact that the amount of research in explainable AI is growing actively, there is no universal consensus on the exact definition of interpretability and its measurement criterion \cite{ref01}. Ruping first noted that interpretability is composed of three goals - accuracy, understandability, and efficiency \cite{ref03}. We argue that fidelity is a better description than accuracy since accuracy is easily confused with the performance evaluation criteria of the original black box model. These three goals are inextricably intertwined and competing with each other, as shown in Figure~\ref{fig:2}. An explainable model with good interpretability should be faithful to the data and the original model, understandable to the observer and graspable in a short time so that the end-users can make wise decisions.
\begin{figure}
\begin{minipage}{1\linewidth}
\centering
\centerline{\includegraphics[width=0.5\textwidth]{figures/process/characteristics.pdf}}
%\centerline{\scriptsize{(b) Decision Tree Expanation}}
%\centerline{Decision Tree Explanation}
\end{minipage}
\caption{\small{The three goals of Interpretability}}
%\vspace{-10pt}
\label{fig:2}
\end{figure}
%This section describes the local interpretable model-agnostic explanation enabled with decision tree model (TLIME).
TLIME has many appealing characteristics, such as interpretable, local fidelity, and model-agnostic. It provides a qualitative understanding of features and predictions. It is challenging, if not impossible, to be utterly faithful to the black box model on a global scale. TLIME takes a feasible approach by approximating it in the vicinity of an instance being predicted. Besides, TLIME, as a model-agnostic interpretation, shows excellent flexibility and capability of explaining any underlying machine learning model.
% \cite{model-agnostic}
%\begin{figure}
%\begin{minipage}{0.6\linewidth}
% \centering
% \centerline{\includegraphics[width=1\textwidth]{figures/process/limelinear}}
% %\centerline{\scriptsize{(a) Sparse Linear Explanation}}
% %\centerline{Sparse Linear Explanation}
%\end{minipage}
%
%\begin{minipage}{0.9\linewidth}
% \centering
% \centerline{\includegraphics[width=1\textwidth]{figures/process/limetree}}
% %\centerline{\scriptsize{(b) Decision Tree Expanation}}
% %\centerline{Decision Tree Explanation}
%\end{minipage}
%\caption{Sparse Linear Explanation and Decision Tree Explanation}
%\label{fig:2}
%\end{figure}
%\begin{figure}
%\begin{minipage}{0.9\linewidth}
% \centering
% \centerline{\includegraphics[width=1\textwidth]{figures/process/limetree}}
% %\centerline{\scriptsize{(b) Decision Tree Expanation}}
% %\centerline{Decision Tree Explanation}
%\end{minipage}
%\caption{Sparse Linear Explanation and Decision Tree Explanation}
%\label{fig:3}
%\end{figure}
%\subsection{Interpretable Image Representations}
\subsection{Explanation System of TLIME}
Considering the poor interpretability and high computational complexity of the pixel-based image representation, we adopt a superpixel based explanation system. Each superpixel, as the basic processing unit, is a group of connected pixels with similar colors or gray levels. Figure~\ref{fig:3} shows the pixel-based image, superpixel image, and superpixel-based explanation. The interpretable representation of an image $x\in{\mathbb{R}^d} $ consisting of $d$ pixels and $d'$ superpixels is a binary vector $x'\in{\{{0,1}\}^{d'}}$ where $1$ indicates the presence of original superpixel and $0$ indicates absence of original superpixel.
%Moreover, we use the classicial felzenszwalb's graph-based segmentation method \cite{fel} to get superpixel image as shown in Figure~\ref{fig:3}.
\begin{figure}
\begin{minipage}{0.32\linewidth}
%\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon}}
\centerline{\scriptsize{(a) Pixel-based image}}
%\centerline{}
\end{minipage}
%\wfill
\begin{minipage}{0.32\linewidth}
%\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon_p}}
\centerline{\scriptsize{(b) Superpixel image}}
%\centerline{}
\end{minipage}
\begin{minipage}{0.32\linewidth}
%\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon_p1}}
\centerline{\scriptsize{(c) Superpixel-based explanation}}
%\centerline{}
\end{minipage}
\caption{\small{Pixel-based image and superpixel image}}
\label{fig:3}
\end{figure}
We denote the original image classification model being explained as $f$, the interpretable decision tree model as $g$, and the locality fidelity loss as $L(f,g,\pi_x)$, which is calculated by the locally weighted square loss:
%\begin{equation}
%\mathcal{l}(f(x),g(x),w(x))=\sum_{}D(f(x),g(x),w(x))\\
%D(f(x),g(x),w(x))=
%
%\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}x)}_{{\sigma_1}i,{\sigma_2}j}=\hat x_{i,j}\omega^{-({\tau_1}i+{\tau_2}j)}, (i,j)\in{\Omega_n}
%\end{equation}
%\begin{equation}
%\L(f,g,\pi_x)=\sum_{z,z'\in Z}D(f(z),g(z'),\pi_x(z))\\
%\end{equation}
%where
%\begin{eqnarray}
%%\centering
%%\L(f,g)=\sum_{z,z'\in Z}D(f(x),g(x),\pi_x(z))\\
%D(f(z),g(z'),\pi_x(z))=
%\left\{
%\begin{array}{lll}
%%$|1-\hat{G}_{i,j}|\leq \delta & (i,j)\in{D_{{\epsilon}^{'}N}}$\\
%%$|\hat{G}_{i,j}|< \delta & (i,j)\in{D_{{\epsilon}^{'}N}}$
%\pi_x(z), & f(z)\ne{g(z')}\\
%0, & f(z)=g(z')
%\end{array}
%\right.
%\end{eqnarray}
\begin{equation}
\L(f,g,\pi_x)=\sum_{z,z'\in Z}e^{(-D(x,z)^2/{\sigma}^2)}(f(z)-g (z'))^2.\\
\end{equation}
%where
%\begin{eqnarray}
%%\centering
%%\L(f,g)=\sum_{z,z'\in Z}D(f(x),g(x),\pi_x(z))\\
%D(f(z),g(z'),\pi_x(z))=
%\left\{
%\begin{array}{lll}
%%$|1-\hat{G}_{i,j}|\leq \delta & (i,j)\in{D_{{\epsilon}^{'}N}}$\\
%%$|\hat{G}_{i,j}|< \delta & (i,j)\in{D_{{\epsilon}^{'}N}}$
%\pi_x(z), & f(z)\ne{g(z')}\\
%0, & f(z)=g(z')
%\end{array}
%\right.
%\end{eqnarray}
The database $Z$ is composed of perturbed samples $z'\in{\{{0,1}\}^{d'}}$ which are sampled around $x'$ by drawing nonzero elements at random. Given a perturbed sample $z'$, we recover the sample in the original representation $z\in {\mathbb{R}^d}$ and get $f(z)$. Moreover, $\pi_x(z)=exp(-D(x,z)^2/{\sigma}^2)$ where distance function $D$ is the $L_2$ distance of image $x$ and $z$ is used to capture locality.
%as shown in Figure~\ref{fig:3}The binary vector $x'$ is the interpretable representation of $z$.
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.6\textwidth]{figures/process/perturbed}}
%\end{minipage}
%\caption{Perturbed samples}
%\label{fig:3}
%\end{figure}
We denote the decision tree explanation produced by TLIME as below:
\begin{equation}
\xi(x)=argmin\quad{L(f,g,\pi_x)+ dep(g)}.
\end{equation}
The depth of decision tree $dep(g)$ is a measure of model complexity. A smaller depth indicates a stronger understandability of model $g$. In order to ensure both local fidelity and understandability, formula (2) minimizes locality-fidelity loss $L(f,g,\pi_x)$ while holding $dep(g)$ low enough. Algorithm 1 shows a simplified workflow diagram of TLIME. Firstly, TLIME gets the superpixel image by using a standard segmentation method. Then the database $Z$ is constructed by running multiple iterations of the perturbed sampling operation. Finally, within the allowable range of prediction error, TLIME gets the minimum depth decision tree by using the CART method.
%For the regression tasks, CART takes a feature and determines which cut-off point minimizes the mean squared error. After determining the best cutoff of each feature, the algorithm selects the feature for splitting that would result in the best partition. Then, CART continues search-and-split recursively in new nodes until a stop criterion is reached.
%
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.8\textwidth]{figures/process/111}}
%\end{minipage}
%\caption{TLIME}
%\label{fig:4}
%\end{figure}
%
%\begin{algorithm}[htb]
%\setstretch{1.35} %ÉèÖþßÓÐÖ¸¶¨µ¯Á¦µÄÏðƤ³¤¶È£¨ÔÐпíµÄ1.35 ±¶£©
%\caption{}
%\label{alg:Framwork}
%\begin{algorithmic}
%\REQUIRE ~~\\
%The set of positive samples for current batch, $P_n$;\\
%The set of unlabelled samples for current batch, $U_n$;\\
%Ensemble of classifiers on former batches, $E_{n-1}$;
%
%
%\ENSURE ~~\\
%Ensemble of classifiers on the current batch, $E_n$;
%Extracting the set of reliable negative and/or positive samples $T_n$ from $U_n$ with help of $P_n$;\\
%\label{ code:fram:extract }
%
%Training ensemble of classifiers $E$ on $T_n \cup P_n$, with help of data in former batches;\\
%\label{code:fram:trainbase}
%
%$E_n=E_{n-1}\cup E$;\\
%\label{code:fram:add}
%
%Classifying samples in $U_n-T_n$ by $E_n$;\\
%\label{code:fram:classify}
%
%Deleting some weak classifiers in $E_n$ so as to keep the capacity of $E_n$;\\
%\label{code:fram:select}
%
%$E_n$;\\
%\end{algorithmic}
%\end{algorithm}
\begin{algorithm}[h]
\caption{\small{Decision Tree based Local interpretable model-agnostic explanation (TLIME)}}
\label{alg::conjugateGradient}
\begin{algorithmic}[1]
\footnotesize
\REQUIRE
Classifier $f$;
Number of samples $N$;
Instance $x$;
Max depth of tree $d$;
\ENSURE
time and prediction error of TLIME;
\STATE get superpixel image $x'$ by segment method;
\STATE initial $Z=\{\}$;
\FOR{$i=1$; $i<N$; $i++$}
\STATE get $z'$ by sampling around $x'$;
\STATE get $f(x')$ by classifier $f$;
\STATE get $z$ by recovering $z'$;
\STATE $Z=Z+(z'_i,f(z_i),\pi_x(z_i))$
\ENDFOR
%\STATE initial $dep=1$;
\FOR{$j=1$; $j<d$ and $error<\delta$; $j++$}
%\STATE get decision tree $g$ by CART algorithm;
\STATE get decision tree $g=CART(Z, maxdepth=j)$
\STATE $error=\|f(x)-g(x')\|$;
\ENDFOR
\STATE output decision tree $g$, time and prediction error;
\end{algorithmic}
\end{algorithm}
%We sample instances around $x'$ by drawing nonzero elements of $x'$ uniformly at random
%
%%: \mathbb{R}^d\to\mathbb{R}$.
%The interpretable model decision tree is represented as $g$,
%
%For image classification, the interpretable representation is a binary vector indicating
%
%For classifier, a image is generally presented as a tensor with three color channels per pixel.
%The explanation produced by TLIME is represented as
%
%(1) for image classification, interpretable representation is a binary vector indicating the super-pixel. Insert the superpixel image and the binary vector.(2) introduce TLIME formula and principle. Insert the perturb image (3) draw algorithm flow chart or block diagram.
\section{Experimental Results}
\label{sec:typestyle}
In this section, TLIME and LIME explain the predictions of RandomForeset Classifier and Google's pre-trained Inception neural network. We compare the experimental results of the two algorithms in terms of understandability, fidelity, and efficiency. %All experiments were carried out on the system that is equiped with 8 Intel Xeon E7-8830 2.13GHz CPU total of 64 cores and 1TB memory. we present numerical experimental results
\subsection{RandomForeset Classifier on MNIST database}
The MNIST database is one of the most common databases used for image classification. It consists of $7\times{10^5}$ small $28\times28$ grayscale images of handwritten digits. In this experiment, the image data is split into $70\%$ as the training set and $30\%$ as the test set. Table~\ref{tab:2} shows the performance of the random forest classifier. For instance $x$, the predicted top $1$ classe is $Seven$ (p=1.0). Figure~\ref{fig:6} shows the decision tree explanations by TLIME.
%For instance $x$, the prediction probability matrix is $f(x)=[0,1.0,0,0,0,0,0,0,0,0,0]$.Some representative images are shown in Figure~\ref{fig:5}.
\begin{table}[h]
\centering\footnotesize
\caption{\small{The performance of randomforeset classifier on MNIST database}}
\begin{tabular}{|c|c|c|c|c|}
\hline
& precision & recall & f1-score & support \bigstrut\\
\hline
weighted avg & 0.95 & 0.95 & 0.95 & 21000
\bigstrut\\
\hline
\end{tabular}%
\label{tab:2}%
\end{table}
%\begin{figure}
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist0}}
% %\centerline{\scriptsize{(1) Original image $\hat{x}(k=2)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist1}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist2}}
% %\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist3}}
% %\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist4}}
% %\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\\
%\vfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist5}}
% %\centerline{\scriptsize{(3) filtered image $\widehat{(G\cdot %P_{\sigma_1,\sigma_2,\tau_1,\tau_2}x)}$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist6}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist7}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist8}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist9}}
% %\centerline{\scriptsize{(4) Subsampled image in frequency domain}}
% %\centerline{}
%\end{minipage}
%\caption{Some representative images in MNIST database}
%\label{fig:5}
%\end{figure}
%\subsubsection{Understandability}
Comparing with LIME, which can only provide a one-shot explanation, the decision tree structure by TLIME provides a more intuitive explanation. Figure~\ref{fig:6} shows that if feature $0$ and feature $3$ exist, then the prediction probability is $1.0$. Moreover, the tree structure can capture the interaction between features in the data. The importance of feature $0$ and $3$ is $0.9416$ and $0.0402$, respectively, which tells us the feature $0$ makes a significant contribution to predicting the outcome. The prediction error is calculated to measure local fidelity. The prediction error of TLIME is $0.0$, showing better fidelity than LIME with an error of $0.0529$.
Efficiency is highly related to the time necessary for a user to grasp the explanation. The runtime of TLIME is $0.0020s$, which is faster than that of LIME - $0.0080s$. Note that the runtime does not include perturbed sampling operation, which takes the same time for LIME and TLIME.%Moreover, the experiment results by running experiments 5000 times are shown in Table~\ref{tab:3}.
%TLIME creates good explanations as the data ends up in distinct groups that are often easy to understand.
\begin{figure*}
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office1}}
%\centerline{\scriptsize{(1) Original image $\hat{x}(k=2)$}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office_12_6}}
%\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office_36_28}}
%\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/tree846_16_29_21}}
%\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
%\centerline{}
\end{minipage}
\\
\vfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=0.9\textwidth]{figures/process/office_class.pdf}}
%\centerline{\scriptsize{(1) Original image and predict probability}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=0.9\textwidth]{figures/process/office_simple_846.pdf}}
%\centerline{\scriptsize{(2) Explaining $table_lamp$}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=0.9\textwidth]{figures/process/office_simple_831.pdf}}
%\centerline{\scriptsize{(3) Explaining $studio_couch$}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.23\linewidth}
\centerline{\includegraphics[width=0.9\textwidth]{figures/process/office_simple_721.pdf}}
%\centerline{\scriptsize{(4) Explaining $pillow$}}
%\centerline{}
\end{minipage}
\small
\caption{\small{Explaining an image classification prediction made by Google's Inception neural network. The top $3$ classes predicted are $table$ $lamp (p=0.4785)$, $studio$ $couch (p=0.4203)$, $pillow (p=0.0227)$}} \small{(a) Original image and prediction probability. (b) Explaining $table$ $lamp$. (c) Explaining $studio$ $couch$. (d) Explaining $pillow$.}
\vspace{-10pt}
\label{fig:7}
\end{figure*}
\begin{figure}
\begin{minipage}{1\linewidth}
\centering
\centerline{\includegraphics[width=0.9\textwidth]{figures/process/mnist_visio_tree0700}}
\end{minipage}
\small
\caption{\small{Explaining an image classification prediction made by randomforeset classifier. The top $1$ class predicted is $seven$ $(p=1.0)$}}
%\vspace{-10pt}
\label{fig:6}
\end{figure}
\begin{table}
\centering\footnotesize
\caption{\small{The prediction probability and prediction errors of TLIME and LIME on Google's Inception neural network}}
\begin{tabular}{|c|c|c|c|c|}
%\begin{tabular}{|c|c|c|c|}
\hline
& Inception prob & pred prob & pred error \bigstrut\\
\hline
TLIME &\multirow{2}{*}{$p(lamp)=0.4785$}& 0.4309 & 0.0476 \bigstrut\\
%\hline
LIME & & 0.6264 & 0.1479 \bigstrut\\
\hline
TLIME &\multirow{2}{*}{$p(couch)=0.4203$}& 0.4682 & 0.0479 \bigstrut\\
%\hline
LIME & & 0.6814 & 0.2611 \bigstrut\\
\hline
TLIME &\multirow{2}{*}{$p(pillow)=0.0227$}& 0.0191 & 0.0036 \bigstrut\\
%\hline
LIME & & 0.0168 & 0.0059 \bigstrut\\
\hline
\end{tabular}
%\vspace{-50pt}
\label{tab:3}
\end{table}
\begin{table}
\centering\footnotesize
\caption{\small{The Time of TLIME and LIME on Inception neural network}}
\begin{tabular}{|c|c|c|c|}
%\begin{tabular}{|c|c|c|c|}
\hline
& $table$ $lamp$ & $studio$ $couch$ & $pillow$ \bigstrut\\
\hline
TLIME & 0.0060s & 0.0030s &0.0090s\bigstrut\\
%\hline
LIME & 0.0289s & 0.0150s & 0.0120s\bigstrut\\
\hline
\end{tabular}
\label{tab:4}
\end{table}
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.9\textwidth]{figures/process/tree1_71_LIME}}
%\end{minipage}
%\caption{Linear explanations by LIME}
%\label{fig:7}
%\end{figure}
%
%\begin{figure*}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.9\textwidth]{figures/process/decision2}}
%\end{minipage}
%\caption{Decision tree explanations}
%\label{fig:7}
%\end{figure*}
%\begin{table}[htbp]
% \centering
% \caption{The performance of randomforeset classifier}
% \begin{tabular}{|c|c|c|c|c|}
% %\begin{tabular}{|c|c|c|c|}
% \hline
% & precision & recall & f1-score & support \bigstrut\\
% \hline
% 0 & 0.96 & 0.98 & 0.97 & 1356
% \bigstrut\\
% \hline
% 1 & 0.97 & 0.98 & 0.97 & 1637
% \bigstrut\\
% \hline
% 2 & 0.93 & 0.95 & 0.94 & 1405
% \bigstrut\\
% \hline
% 3 & 0.92 & 0.93 & 0.92 & 1404
% \bigstrut\\
% \hline
% 4 & 0.93 & 0.95 & 0.94 & 1373
% \bigstrut\\
% \hline
% 5 & 0.92 & 0.92 & 0.92 & 1212
% \bigstrut\\
% \hline
% 6 & 0.97 & 0.97 & 0.97 & 1374
% \bigstrut\\
% \hline
% 7 & 0.97 & 0.96 & 0.96 & 1459
% \bigstrut\\
% \hline
% 8 & 0.95 & 0.90 & 0.92 & 1417
% \bigstrut\\
% \hline
% 9 & 0.94 & 0.92 & 0.93 & 1363
% \bigstrut\\
% \hline
% micro avg & 0.95 & 0.95 & 0.95 & 14000
% \bigstrut\\
% \hline
% macro avg & 0.94 & 0.94 & 0.94 & 14000
% \bigstrut\\
% \hline
% weighted avg & 0.95 & 0.95 & 0.95 & 14000
% \bigstrut\\
% \hline
% \end{tabular}%
% \label{tab:3}%
%\end{table}
%\subsubsection{Fidelity}
%If fidelity is not achieved, the explanation is useless, as it explains nothing that has connection to the data and thus to the reality.
%\begin{table}[htbp]
% \centering
% \caption{The predict error of LIME and TLIME}
% \begin{tabular}{|c|c|c|}
% %\begin{tabular}{|c|c|c|c|}
% \hline
% & LIME & TLIME \bigstrut\\
% \hline
% predict error & 0.074 & 0.012
% \bigstrut\\
% \hline
% \end{tabular}%
% \label{tab:3}%
%\end{table}
%\subsubsection{Efficiency}
%Moreover, the runtime by running experiments 5000 times is shown in Table~\ref{tab:4}.
%\begin{table}[htbp]
% \centering
% \caption{The Runtime of LIME and TLIME}
% \begin{tabular}{|c|c|c|}
% %\begin{tabular}{|c|c|c|c|}
% \hline
% & LIME & TLIME \bigstrut\\
% \hline
% runtime & 0.0243s & 0.0062s
% \bigstrut\\
% \hline
% \end{tabular}%
% \label{tab:4}%
%\end{table}
\subsection{Google's Inception neural network on Image-net database}
We explain the prediction of Google's pre-trained Inception neural network on the image shown in Figure~\ref{fig:7}a. The top $5$ predicted classes are listed. Figures~\ref{fig:7}b, \ref{fig:7}c, \ref{fig:7}d show the superpixels explainations for the top $3$ predicted classes: $table$ $lamp (p=0.4785)$, $studio$ $couch (p=0.4203)$ and $pillow (p=0.0227)$ respectively. The prediction provides reasonable insight into what the neural network picks upon for each of the classes. This kind of explanation enhances trust in the classifier. Moreover, Table~\ref{tab:3} lists the prediction errors of TLIME and LIME. Table~\ref{tab:4} lists the runtime of TLIME and LIME. We can conclude from the above results that under less time, TLIME not only has a better understandability but also has a higher fidelity than LIME.
%Figure~\ref{fig:7} show the superpixels explanations for the top $3$ predicted classes.
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.9\textwidth]{figures/process/incep.pdf}}
%\end{minipage}
%\caption{The top 5 prediction by Google's Inception neural network}
%\label{fig:7}
%\end{figure}
%\begin{figure*}
%\begin{minipage}{0.32\linewidth}
% %\centering
% \centerline{\includegraphics[width=1.0\textwidth]{figures/process/incep.pdf}}
% \centerline{\scriptsize{(a)Original Image}}
% %\centerline{}
%\end{minipage}
%%\wfill
%\begin{minipage}{0.32\linewidth}
% %\centering
% \centerline{\includegraphics[width=1.0\textwidth]{figures/process/mnist_visio_tree47_01.pdf}}
% \centerline{\scriptsize{(b)Explaining African chameleon}}
% %\centerline{}
%\end{minipage}
%\begin{minipage}{0.32\linewidth}
% %\centering
% \centerline{\includegraphics[width=1.0\textwidth]{figures/process/mnist_visio_tree47_01.pdf}}
% \centerline{\scriptsize{(c)Explaining African chameleon}}
% %\centerline{}
%\end{minipage}
%\caption{Explaining an image classification prediction made by Google's Inception neural network. The top 2 classes predicted are "African chameleon"(p=0.99),"Fficdal"(p=0.2)}
%\label{fig:3}
%\end{figure*}
%\begin{figure*}
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/office_class.pdf}}
% %\centerline{\scriptsize{(1) Original image $\hat{x}(k=2)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist_visio_tree846.pdf}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist_visio_tree831.pdf}}
% %\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/mnist_visio_tree721_1}}
% %\centerline{\scriptsize{(2) permuted image %$\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\\
%\vfill
%\begin{minipage}{0.19\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/office1}}
% %\centerline{\scriptsize{(3) filtered image $\widehat{(G\cdot %P_{\sigma_1,\sigma_2,\tau_1,\tau_2}x)}$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/tree846_36_20}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/tree846_13_6}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\hfill
%\begin{minipage}{0.24\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/tree846_16_29_21}}
% %\centerline{\scriptsize{(2) permuted image $\widehat{(P_{\sigma_1,\sigma_2,\tau_1,\tau_2}}x)$}}
% %\centerline{}
%\end{minipage}
%\caption{Some representative images in MNIST database}
%\label{fig:7}
%\end{figure*}
%\begin{figure}
%\begin{minipage}{1\linewidth}
% \centering
% \centerline{\includegraphics[width=0.9\textwidth]{figures/process/mnist_visio_tree47_01.pdf}}
%\end{minipage}
%\caption{Decision tree explanations by TLIME}
%\label{fig:8}
%\end{figure}
%\section{Conclusion}
%\label{sec:majhead}
%\%begin{figure}
%\begin{minipage}{0.45\linewidth}
% %\centering
% \centerline{\includegraphics[width=1.0\textwidth]{figures/process/tree47_022_28_30}}
% \centerline{\scriptsize{(a)}}
% %\centerline{}
%\end{minipage}
%%\wfill
%\begin{minipage}{0.45\linewidth}
% %\centering
% \centerline{\includegraphics[width=1.0\textwidth]{figures/process/mnist_visio_tree471.pdf}}
% \centerline{\scriptsize{(b)}}
% %\centerline{}
%\end{minipage}
%\caption{Felzenszwalbs method}
%\label{fig:3}
%\end{figure}
\section{Conclusion}
We propose a decision tree-based local interpretable model-agnostic explanation (TLIME) for improving explainable AI. The goal of TLIME is to construct an interpretable decision tree model over the interpretable representation that is locally faithful to the oringal classifier. We compare TLIME and LIME in explaining the predictions of RandomForeset Classifier and Google's pre-trained Inception neural network. Experimental results have shown that TLIME exhibits a better understandability and higher fidelity than LIME using less process time, which covers the ingredients of an ideal explainable AI model - understandability, fidelity, and efficiency.
%Moreover, the tree structure can capture interactions between features in the data.
% Below is an example of how to insert images. Delete the ``\vspace'' line,
% uncomment the preceding line ``\centerline...'' and replace ``imageX.ps''
% with a suitable PostScript file name.
% -------------------------------------------------------------------------
%\begin{figure}[htb]
%
%\begin{minipage}[b]{1.0\linewidth}
% \centering
% \centerline{\includegraphics[width=8.5cm]{image1}}
%% \vspace{2.0cm}
% \centerline{(a) Result 1}\medskip
%\end{minipage}
%%
%\begin{minipage}[b]{.48\linewidth}
% \centering
% \centerline{\includegraphics[width=4.0cm]{image3}}
%% \vspace{1.5cm}
% \centerline{(b) Results 3}\medskip
%\end{minipage}
%\hfill
%\begin{minipage}[b]{0.48\linewidth}
% \centering
% \centerline{\includegraphics[width=4.0cm]{image4}}
%% \vspace{1.5cm}
% \centerline{(c) Result 4}\medskip
%\end{minipage}
%%
%\caption{Example of placing a figure with experimental results.}
%\label{fig:res}
%%
%\end{figure}
% To start a new column (but not a new page) and help balance the last-page
% column length use \vfill\pagebreak.
% -------------------------------------------------------------------------
%\vfill
%\pagebreak
%\section{REFERENCES}
%\label{sec:refs}
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\newpage
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}
%\begin{thebibliography}{9}
%\end{thebibliography}
\end{document}
% Template for ICASSP-2020 paper; to be used with:
% spconf.sty - ICASSP/ICIP LaTeX style file, and
% IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}
%\usepackage[ruled]{algorithm2e}
\usepackage{threeparttable}
\usepackage{dcolumn}
\usepackage{booktabs}
\usepackage{amsmath,graphicx}
\usepackage{url}
\usepackage{cite}
\usepackage{psfrag}
\usepackage{amsfonts,amssymb}
\usepackage{amsbsy}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{array}
\usepackage{multirow}
\usepackage{bm}
\usepackage{subfigure}
\usepackage{verbatim}
\usepackage{colortbl}
\usepackage{stfloats}
%\usepackage{algorithmicx}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{setspace}
%\usepackage{algorithm2e,setspace}
\usepackage{bigstrut}
\usepackage{amsmath}
\usepackage{mathtools}
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}
% Title.
% ------
\title{Explaining the Predictions of Any Image Classifier via Decision Trees}
%
% Single address.
% ---------------
%\name{Author(s) Name(s)\thanks{Thanks to XYZ agency for funding.}}
%\address{Author Affiliation(s)}
\name{Sheng Shi$^{\dagger}$ \qquad Xinfeng Zhang$^{\star}$ \qquad Wei Fan$^{\dagger}$}
\address{$^{\dagger}$ AI Laboratory, Lenovo Research, Beijing 100094, China \\
$^{\star}$ University of Chinese Academy of Sciences, Beijing 100049, China}
%
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Despite outstanding contribution to the significant progress of Artificial Intelligence (AI), deep learning models remain mostly black boxes, which are extremely weak in explainability of the reasoning process and prediction results. Explainability is not only a gateway between AI and society but also a powerful tool to detect flaws in the model and biases in the data. Local Interpretable Model-agnostic Explanation (LIME) is a recent approach that uses an interpretable model to form a local explanation for the individual prediction result. The current implementation of LIME adopts the linear regression as its interpretable function. However, being so restricted and usually over-simplifying the relationships, linear models fail in situations where nonlinear associations and interactions exist among features and prediction results. This paper implements a decision Tree-based LIME approach, which uses a decision tree model to form an interpretable representation that is locally faithful to the original model. Tree-LIME approach can capture nonlinear interactions among features in the data and creates plausible explanations. Various experiments show that the Tree-LIME explanation of multiple black-box models can achieve more reliable performance in terms of understandability, fidelity, and efficiency.
\end{abstract}
%
\begin{keywords}
Explainable AI, Interpretable Model, Decision Tree, LIME, Local Fidelity
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
In recent years, the fast-growing computing power, enormous consumer and commercial data, and emerging advanced machine learning algorithms jointly stimulate the prosperous of AI \cite{AI01}\cite{AI02}, which has gone from a science-fiction dream to a critical part of our daily life. Compared to traditional machine learning methods, deep learning has achieved superior performance in perception tasks such as object detection and classification. However, because of the nested non-linear structure, deep learning models usually remain black boxes that are particularly weak in the explainability of the reasoning process and prediction results. In many real-world mission-critical applications, transparency of deep learning models and explainability of the model outputs are essential and necessary in their real deployment process.
Explanable AI is not only a gateway between AI and society but also a powerful tool to detect flaws in the model and biases in the data. The development of techniques on explainability and transparency of deep learning models has recently received much attention in the research community \cite{One01}\cite{One02}\cite{One03}\cite{One04}. The relevant research roughly falls into two categories: global explainability and local explainability. Global explainability aims at making the reasoning process wholly transparent and comprehensive \cite{global03}\cite{global04}, while local explainability focuses on extracting input regions that are highly sensitive to the network output to provide explanations for each decision \cite{local01}\cite{local02}\cite{local03}\cite{local04}.
An effective way to achieve explainability is to use a light-weight function family to create interpretable models. Local interpretable model-agnostic explanations (LIME) identify an interpretable model over the human-interpretable representation that is locally faithful to the original model \cite{local01}. The current implementation of LIME in \cite{local01} adopts the linear regression as its interpretable function, which represents the prediction as a linear combination of a few selected features to make the prediction process transparent (Linear-LIME). However, being so restricted and usually over-simplifying the relationships, linear regression models fail in some situations where non-linear associations and interactions exist among features and prediction results %\cite{ref01}.
The LIME framework in \cite{local01} supports the exploration of a variety of explanation families, such as linear regression and decision trees. In this paper, we implement a Decision Tree-based Local Interpretable Model-agnostic Explanation (Tree-LIME). The decision tree structure creates good explanations as the data ends up in distinct groups that are often easy to understand. Moreover, the tree structure can capture interactions between features in the data. We perform various experiments on explaining two black-box models, the random-forest classifier and Google's pre-trained Inception neural network\cite{Inception}. The results show that Tree-LIME explanations achieve more reliable performance than Linear-LIME in terms of understandability, fidelity, and efficiency.
\section{Interpretable models}
\label{sec:format}
Using a subset of algorithms from a light-weight function family to create interpretable models is an effective way to achieve interpretability. In this section, we analyze two representative interpretable models - the linear regression model and the decision tree model. Table~\ref{tab:1} shows the properties of two interpretable models. The linear regression displays the prediction as a linear combination of features, while the decision tree represents the reasoning process in a hierarchical structure, which is suitable for capturing the nonlinear association between features and predictions. The monotonicity constraint shown in both models is necessary to ensure the consistency between a feature and the target outcome. Moreover, the decision tree model can automatically capture the diverse interactions between features to predict the target outcome, applicable to both classification and regression tasks. %In this paper, we mainly focus on the image classifier model.
\begin{table*}
\centering\footnotesize
\caption{\small{The properties of linear regression model and decision tree model}}
\begin{tabular}{|c|c|c|c|c|}
%\begin{tabular}{|c|c|c|c|}
\hline
Models & Linearity & Monotonicity & Feature Interaction & Task \bigstrut\\
\hline
Line regression & Yes & Yes & No & Regression
\bigstrut\\
\hline
Decision tree & No & Some & Yes & Classification, Regression\bigstrut\\
\hline
\end{tabular}%
\label{tab:1}%
\end{table*}
Depending on the different criteria, various algorithms are capable of constructing a decision tree. The CART \cite{ref02} is the most popular algorithm which can handle both classification and regression tasks. In this paper, we mainly construct regression decision trees to explain the prediction probability of the image classifier. Figure~\ref{fig:1} illustrates a simple regression tree to explain image classification prediction made by Google's Inception neural network. The predicted top $1$ class label is $African$ $chanmeleon$ $(p=0.9935)$. The highlighted superpixels give intuition as to why the model would choose that class. The decision tree shows that if feature $28$, $22$, and $30$ exist, then the prediction probability is $0.991$, which is the mean value of the instances $y$ in this node. Moreover, The importance of the three features is $0.7164$, $0.0709$, and $0.0259$, showing the contribution of the three features in improving the variance.
%\end{figure}
\begin{figure}
\begin{minipage}{1\linewidth}
\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon_simple.pdf}}
\end{minipage}
\caption{\small{A simple regression decision tree explains the image classification results given by Google's Inception neural network. The top $1$ output class label is $African$ $chanmeleon$ with the prediction probability $(p=0.9935)$.}}
\vspace{-10pt}
\label{fig:1}
\end{figure}
\section{The Tree-LIME Approach}
\label{sec:pagestyle}
In this section, we first present the interpretable image representations and the decision tree-based LIME approach. Then we discuss the characteristics of Tree-LIME.
%\subsection{Interpretable Image Representations}
\subsection{Explanation System of Tree-LIME}
Considering the poor interpretability and high computational complexity of the pixel-based image representation, we adopt a superpixel based explanation system. Each superpixel, as the basic processing unit, is a group of connected pixels with similar colors or gray levels. Figure~\ref{fig:3} shows the pixel-based image, superpixel image, and superpixel-based explanation. The interpretable representation of an image $x\in{\mathbb{R}^d} $ consisting of $d$ pixels and $d'$ superpixels is a binary vector $x'\in{\{{0,1}\}^{d'}}$ where $1$ indicates the presence of original superpixel and $0$ indicates absence of original superpixel.
%Moreover, we use the classicial felzenszwalb's graph-based segmentation method \cite{fel} to get superpixel image as shown in Figure~\ref{fig:3}.
\begin{figure}
\begin{minipage}{0.32\linewidth}
%\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon.pdf}}
\centerline{\scriptsize{(a) Pixel-based image}}
%\centerline{}
\end{minipage}
%\wfill
\begin{minipage}{0.32\linewidth}
%\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon_p.pdf}}
\centerline{\scriptsize{(b) Superpixel image}}
%\centerline{}
\end{minipage}
\begin{minipage}{0.32\linewidth}
%\centering
\centerline{\includegraphics[width=1.0\textwidth]{figures/process/chanmeleon_p1.pdf}}
\centerline{\scriptsize{(c) Superpixel-based explanation}}
%\centerline{}
\end{minipage}
\caption{\small{Pixel-based image and superpixel image}}
\label{fig:3}
\end{figure}
We denote the original image classification model being explained as $f$, the interpretable decision tree model as $g$, and the locality fidelity loss as $L(f,g,\pi_x)$, which is calculated by the locally weighted square loss:
\begin{equation}
\L(f,g,\pi_x)=\sum_{z,z'\in Z}e^{(-D(x,z)^2/{\sigma}^2)}(f(z)-g (z'))^2.\\
\end{equation}
The database $Z$ is composed of perturbed samples $z'\in{\{{0,1}\}^{d'}}$ which are sampled around $x'$ by drawing nonzero elements at random. Given a perturbed sample $z'$, we recover the sample in the original representation $z\in {\mathbb{R}^d}$ and get $f(z)$. Moreover, $\pi_x(z)=exp(-D(x,z)^2/{\sigma}^2)$ where distance function $D$ is the $L_2$ distance of image $x$ and $z$ is used to capture locality.
We denote the decision tree explanation produced by Tree-LIME as below:
\begin{equation}
\xi(x)=argmin\quad{L(f,g,\pi_x)+ dep(g)}.
\end{equation}
The depth of decision tree $dep(g)$ is a measure of model complexity. A smaller depth indicates a stronger understandability of model $g$. In order to ensure both local fidelity and understandability, formula (2) minimizes locality-fidelity loss $L(f,g,\pi_x)$ while holding $dep(g)$ low enough. Algorithm 1 shows a simplified workflow diagram of Tree-LIME. Firstly, Tree-LIME gets the superpixel image by using a standard segmentation method. Then the database $Z$ is constructed by running multiple iterations of the perturbed sampling operation. Finally, within the allowable range of prediction error, Tree-LIME gets the minimum depth decision tree by using the CART method.
\begin{algorithm}[h]
\caption{\small{Decision Tree based Local interpretable model-agnostic explanation (Tree-LIME)}}
\label{alg::conjugateGradient}
\begin{algorithmic}[1]
\footnotesize
\REQUIRE
Classifier $f$,
Number of samples $N$,
Instance $x$,
Max depth of tree $d$
\ENSURE
time and prediction error of Tree-LIME
\STATE get superpixel image $x'$ by segment method
\STATE initial $Z=\{\}$
\FOR{$i=1$; $i<N$; $i++$}
\STATE get $z'$ by sampling around $x'$
\STATE get $f(x')$ by classifier $f$
\STATE get $z$ by recovering $z'$
\STATE $Z \leftarrow Z \cup (z'_i,f(z_i),\pi_x(z_i))$
\ENDFOR
%\STATE initial $dep=1$;
\FOR{$j=1$; $j<d$ and $error<\delta$; $j++$}
%\STATE get decision tree $g$ by CART algorithm;
\STATE get decision tree $g=CART(Z, maxdepth=j)$
\STATE $error=\|f(x)-g(x')\|$
\ENDFOR
\STATE output decision tree $g$, time and prediction error
\end{algorithmic}
\end{algorithm}
\subsection{Characteristics of Tree-LIME}
Despite the fact that the amount of research in explainable AI is growing actively, there is no universal consensus on the exact definition of interpretability and its measurement criterion \cite{ref01}. Ruping first noted that interpretability is composed of three goals - accuracy, understandability, and efficiency \cite{ref03}. We argue that fidelity is a better description than accuracy since accuracy is easily confused with the performance evaluation criteria of the original black box model. These three goals are inextricably intertwined and competing with each other, as shown in Figure~\ref{fig:2}. An explainable model with good interpretability should be faithful to the data and the original model, understandable to the observer and graspable in a short time so that the end-users can make wise decisions.
\begin{figure}
\begin{minipage}{1\linewidth}
\centering
\centerline{\includegraphics[width=0.5\textwidth]{figures/process/characteristics.pdf}}
%\centerline{\scriptsize{(b) Decision Tree Expanation}}
%\centerline{Decision Tree Explanation}
\end{minipage}
\caption{\small{The three goals of Interpretability}}
%\vspace{-10pt}
\label{fig:2}
\end{figure}
%This section describes the local interpretable model-agnostic explanation enabled with decision tree model (TLIME).
Tree-LIME has many appealing characteristics, such as interpretable, local fidelity, and model-agnostic. It provides a qualitative understanding of features and predictions. It is challenging, if not impossible, to be utterly faithful to the black box model on a global scale. Tree-LIME takes a feasible approach by approximating it in the vicinity of an instance being predicted. Besides, Tree-LIME, as a model-agnostic interpretation, shows excellent flexibility and capability of explaining any underlying machine learning model.
\section{Experimental Results}
\label{sec:typestyle}
In this section, Tree-LIME and Linear-LIME \cite{local01} explain the predictions of RandomForeset Classifier and Google's pre-trained Inception neural network. We compare the experimental results of the two algorithms in terms of understandability, fidelity, and efficiency. %All experiments were carried out on the system that is equiped with 8 Intel Xeon E7-8830 2.13GHz CPU total of 64 cores and 1TB memory. we present numerical experimental results
\subsection{RandomForeset Classifier on MNIST database}
The MNIST database is one of the most common databases used for image classification. It consists of $7\times{10^5}$ small $28\times28$ grayscale images of handwritten digits. In this experiment, the image data is split into $70\%$ as the training set and $30\%$ as the test set. Table~\ref{tab:2} shows the performance of the random forest classifier. For instance $x$, the predicted top $1$ classe is $Seven$ (p=1.0). Figure~\ref{fig:6} shows the decision tree explanations by Tree-LIME.
%For instance $x$, the prediction probability matrix is $f(x)=[0,1.0,0,0,0,0,0,0,0,0,0]$.Some representative images are shown in Figure~\ref{fig:5}.
\begin{table}[h]
\centering\footnotesize
\caption{\small{The performance of randomforeset classifier on MNIST database}}
\begin{tabular}{|c|c|c|c|c|}
\hline
& precision & recall & f1-score & support \bigstrut\\
\hline
weighted avg & 0.95 & 0.95 & 0.95 & 21000
\bigstrut\\
\hline
\end{tabular}%
\label{tab:2}%
\end{table}
%\subsubsection{Understandability}
Comparing with Linear-LIME, which can only provide a one-shot explanation, the decision tree structure by Tree-LIME provides a more intuitive explanation. Figure~\ref{fig:6} shows that if feature $0$ and feature $3$ exist, then the prediction probability is $1.0$. Moreover, the tree structure can capture the interaction between features in the data. The importance of feature $0$ and $3$ is $0.9416$ and $0.0402$, respectively, which tells us the feature $0$ makes a significant contribution to predicting the outcome. The prediction error is calculated to measure local fidelity. The prediction error of Tree-LIME is $0.0$, showing better fidelity than Linear-LIME with an error of $0.0529$.
Efficiency is highly related to the time necessary for a user to grasp the explanation. The runtime of CART in Tree-LIME is $0.0020s$, which is faster than that of K-LASS0 in Linear-LIME \cite{local01} - $0.0080s$.%Moreover, the experiment results by running experiments 5000 times are shown in Table~\ref{tab:3}.
%TLIME creates good explanations as the data ends up in distinct groups that are often easy to understand.
\begin{figure}
\begin{minipage}{1\linewidth}
\centering
\centerline{\includegraphics[width=0.9\textwidth]{figures/process/mnist_visio_tree0700.pdf}}
\end{minipage}
\small
\caption{\small{Explaining an image classification prediction made by randomforeset classifier. The top $1$ class predicted is $seven$ $(p=1.0)$}}
%\vspace{-10pt}
\label{fig:6}
\end{figure}
\begin{table}
\centering\footnotesize
\caption{\small{The prediction probability and prediction errors of Linear-LIME and Tree-LIME on Google's Inception neural network}}
\begin{tabular}{|c|c|c|c|c|}
%\begin{tabular}{|c|c|c|c|}
\hline
& Inception prob & pred prob & pred error \bigstrut\\
\hline
Linear-LIME & \multirow{2}{*}{$p(lamp)=0.4785$}& 0.6264 & 0.1479 \bigstrut\\
%\hline
Tree-LIME & & 0.4309 & 0.0476 \bigstrut\\
\hline
Linear-LIME &\multirow{2}{*}{$p(couch)=0.4203$} & 0.6814 & 0.2611 \bigstrut\\
%\hline
Tree-LIME & & 0.4682 & 0.0479 \bigstrut\\
\hline
Linear-LIME &\multirow{2}{*}{$p(pillow)=0.0227$} & 0.0168 & 0.0059 \bigstrut\\
%\hline
Tree-LIME & & 0.0191 & 0.0036 \bigstrut\\
\hline
\end{tabular}
%\vspace{-50pt}
\label{tab:3}
\end{table}
\begin{table}
\centering\footnotesize
\caption{\small{The Time of Linear-LIME and Tree-LIME on Inception neural network}}
\begin{tabular}{|c|c|c|c|}
%\begin{tabular}{|c|c|c|c|}
\hline
& $table$ $lamp$ & $studio$ $couch$ & $pillow$ \bigstrut\\
\hline
%\hline
K-LASS0 in Linear-LIME & 0.0289s & 0.0150s & 0.0120s\bigstrut\\
\hline
CART in Tree-LIME & 0.0060s & 0.0030s &0.0090s\bigstrut\\
\hline
\end{tabular}
\label{tab:4}
\end{table}
\subsection{Google's Inception neural network on Image-net database}
We explain the prediction of Google's pre-trained Inception neural network on the image shown in Figure~\ref{fig:5}(a). Figures~\ref{fig:5}(b), \ref{fig:5}(c) and \ref{fig:5}(d) show the superpixels explainations for the top $3$ predicted classes: $table$ $lamp (p=0.4785)$, $studio$ $couch (p=0.4203)$ and $pillow (p=0.0227)$ respectively. Due to space constraints, Figure~\ref{fig:6} shows the the explainations of $table$ $lamp$ and $studio$ $couch$ by Tree-LIME. The prediction provides reasonable insight into what the neural network picks upon for each of the classes. This kind of explanation enhances trust in the classifier. Moreover, Table~\ref{tab:3} lists the prediction errors of Tree-LIME and Linear-LIME. The prediction error shows that Tree-LIME has a better fidelity than Linear-LIME. Table~\ref{tab:4} lists the runtime of CART in Tree-LIME and runtime of K-LASSO in Linear-LIME.
\begin{figure}
\begin{minipage}{0.24\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office1.pdf}}
\centerline{\scriptsize{(a)}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.24\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office_12_6.pdf}}
\centerline{\scriptsize{(b)}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.24\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office_36_28.pdf}}
\centerline{\scriptsize{(c)}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.24\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/tree846_16_29_21.pdf}}
\centerline{\scriptsize{(d)}}
%\centerline{}
\end{minipage}
\vfill
\caption{\small{Explaining an image classification prediction made by Google's Inception neural network using Tree-LIME. The top $3$ classes predicted are $table$ $lamp (p=0.4785)$, $studio$ $couch (p=0.4203)$, $pillow (p=0.0227)$}: (a) Original image. (b) Explaining $table$ $lamp$. (c) Explaining $studio$ $couch$. (d) Explaining $pillow$.}
\vspace{-10pt}
\label{fig:5}
\end{figure}
\begin{figure}
\begin{minipage}{0.49\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office_simple_846.pdf}}
\centerline{\scriptsize{(a) Explaining $table$ $lamp$}}
%\centerline{}
\end{minipage}
\hfill
\begin{minipage}{0.49\linewidth}
\centerline{\includegraphics[width=1\textwidth]{figures/process/office_simple_831.pdf}}
\centerline{\scriptsize{(b) Explaining $studio$ $couch$}}
%\centerline{}
\end{minipage}
%\hfill
%\begin{minipage}{0.32\linewidth}
% \centerline{\includegraphics[width=1\textwidth]{figures/process/office_simple_721.pdf}}
% \centerline{\scriptsize{(c)}}
%\centerline{}
%\end{minipage}
%\small
\caption{\small{Explaining an image classification prediction made by Google's Inception neural network using Tree-LIME.}}
\vspace{-10pt}
\label{fig:6}
\end{figure}
%Figure~\ref{fig:7} show the superpixels explanations for the top $3$ predicted classes.
%\begin{figure}
We can conclude from the above results that under less time, Tree-LIME not only has a better understandability but also has a higher fidelity than Linear-LIME.
\section{Conclusion}
The LIME framework in \cite{local01} supports the exploration of a variety of explanation families. The current implementation of LIME adopts the linear regression as its interpretable function (Linear-LIME). In this paper, we implement a decision tree-based local interpretable model-agnostic explanation (Tree-LIME). The goal of Tree-LIME is to construct an interpretable decision tree model over the interpretable representation that is locally faithful to the oringal classifier. We compare Tree-LIME and Linear-LIME in explaining the predictions of RandomForeset Classifier and Google's pre-trained Inception neural network. Experimental results have shown that Tree-LIME exhibits a better understandability and higher fidelity than Linear-LIME using less process time, which covers the ingredients of an ideal explainable AI model - understandability, fidelity, and efficiency.
%Moreover, the tree structure can capture interactions between features in the data.
\newpage
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}
%\begin{thebibliography}{9}
%\end{thebibliography}
\end{document}
\begin{thebibliography}{10}
\bibitem{AI01}
R.~{Girshick},
\newblock ``Fast r-cnn,''
\newblock in {\em 2015 IEEE International Conference on Computer Vision
(ICCV)}, Dec 2015, pp. 1440--1448.
\bibitem{AI02}
S.~{Ren}, K.~{He}, R.~{Girshick}, and J.~{Sun},
\newblock ``Faster r-cnn: Towards real-time object detection with region
proposal networks,''
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
vol. 39, no. 6, pp. 1137--1149, June 2017.
\bibitem{One01}
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz,
\newblock ``Identifying unknown unknowns in the open world: Representations and
policies for guided exploration,''
\newblock in {\em Proceedings of the Thirty-First {AAAI} Conference on
Artificial Intelligence, February 4-9, 2017, San Francisco, California,
{USA.}}, 2017, pp. 2124--2132.
\bibitem{One02}
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard~H. Hovy, and Eric~P. Xing,
\newblock ``Harnessing deep neural networks with logic rules,''
\newblock in {\em Proceedings of the 54th Annual Meeting of the Association for
Computational Linguistics, {ACL} 2016, August 7-12, 2016, Berlin, Germany,
Volume 1: Long Papers}, 2016.
\bibitem{One03}
Quanshi Zhang, Ying~Nian Wu, and Song{-}Chun Zhu,
\newblock ``Interpretable convolutional neural networks,''
\newblock in {\em 2018 {IEEE} Conference on Computer Vision and Pattern
Recognition, {CVPR} 2018, Salt Lake City, UT, USA, June 18-22, 2018}, 2018,
pp. 8827--8836.
\bibitem{One04}
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying~Nian Wu, and Song{-}Chun Zhu,
\newblock ``Interpreting {CNN} knowledge via an explanatory graph,''
\newblock in {\em Proceedings of the Thirty-Second {AAAI} Conference on
Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February
2-7, 2018}, 2018, pp. 4454--4463.
\bibitem{global03}
Yao Lu,
\newblock ``Unsupervised learning on neural network outputs: With application
in zero-shot learning,''
\newblock in {\em Proceedings of the Twenty-Fifth International Joint
Conference on Artificial Intelligence, {IJCAI} 2016, New York, NY, USA, 9-15
July 2016}, 2016, pp. 3432--3438.
\bibitem{global04}
Mathieu Aubry and Bryan~C. Russell,
\newblock ``Understanding deep features with computer-generated imagery,''
\newblock in {\em 2015 {IEEE} International Conference on Computer Vision,
{ICCV} 2015, Santiago, Chile, December 7-13, 2015}, 2015, pp. 2875--2883.
\bibitem{local01}
Marco~T{\'{u}}lio Ribeiro, Sameer Singh, and Carlos Guestrin,
\newblock ``Why should {I} trust you?: Explaining the predictions of any
classifier,''
\newblock in {\em Proceedings of the 22nd {ACM} {SIGKDD} International
Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA,
August 13-17, 2016}, 2016, pp. 1135--1144.
\bibitem{local02}
Marco~T{\'{u}}lio Ribeiro, Sameer Singh, and Carlos Guestrin,
\newblock ``Model-agnostic interpretability of machine learning,''
\newblock {\em CoRR}, vol. abs/1606.05386, 2016.
\bibitem{local03}
Ruth~C. Fong and Andrea Vedaldi,
\newblock ``Interpretable explanations of black boxes by meaningful
perturbation,''
\newblock in {\em {IEEE} International Conference on Computer Vision, {ICCV}
2017, Venice, Italy, October 22-29, 2017}, 2017, pp. 3449--3457.
\bibitem{local04}
Ramprasaath~R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam,
Devi Parikh, and Dhruv Batra,
\newblock ``Grad-cam: Visual explanations from deep networks via gradient-based
localization,''
\newblock in {\em {IEEE} International Conference on Computer Vision, {ICCV}
2017, Venice, Italy, October 22-29, 2017}, 2017, pp. 618--626.
\bibitem{Inception}
C.~{Szegedy}, {Wei Liu}, {Yangqing Jia}, P.~{Sermanet}, S.~{Reed},
D.~{Anguelov}, D.~{Erhan}, V.~{Vanhoucke}, and A.~{Rabinovich},
\newblock ``Going deeper with convolutions,''
\newblock in {\em 2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, June 2015, pp. 1--9.
\bibitem{ref02}
Trevor~Hastie Jerome~Friedman and Robert Tibshirani,
\newblock ``The elements of statistical learning: Data mining, inference and
prediction,''
\newblock in {\em https://web.stanford.edu/~hastie/ElemStatLearn/}, 2009.
\bibitem{ref01}
Christoph Molnar,
\newblock ``Interpretable machine learning: A guide for making black box models
explainable,''
\newblock in {\em Lulu, 1st edition, March 24, 2019; eBook}, 2019.
\bibitem{ref03}
S.~Ruping,
\newblock ``Learning interpretable models (phd thesis),''
\newblock in {\em Technical University of Dortmund}, 2006.
\end{thebibliography}
\begin{thebibliography}{10}
\bibitem{AI01}
R.~{Girshick},
\newblock ``Fast r-cnn,''
\newblock in {\em 2015 IEEE International Conference on Computer Vision
(ICCV)}, Dec 2015, pp. 1440--1448.
\bibitem{AI02}
S.~{Ren}, K.~{He}, R.~{Girshick}, and J.~{Sun},
\newblock ``Faster r-cnn: Towards real-time object detection with region
proposal networks,''
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
vol. 39, no. 6, pp. 1137--1149, June 2017.
\bibitem{One01}
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz,
\newblock ``Identifying unknown unknowns in the open world: Representations and
policies for guided exploration,''
\newblock in {\em Proceedings of the Thirty-First {AAAI} Conference on
Artificial Intelligence, February 4-9, 2017, San Francisco, California,
{USA.}}, 2017, pp. 2124--2132.
\bibitem{One02}
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard~H. Hovy, and Eric~P. Xing,
\newblock ``Harnessing deep neural networks with logic rules,''
\newblock in {\em Proceedings of the 54th Annual Meeting of the Association for
Computational Linguistics, {ACL} 2016, August 7-12, 2016, Berlin, Germany,
Volume 1: Long Papers}, 2016.
\bibitem{One03}
Quanshi Zhang, Ying~Nian Wu, and Song{-}Chun Zhu,
\newblock ``Interpretable convolutional neural networks,''
\newblock in {\em 2018 {IEEE} Conference on Computer Vision and Pattern
Recognition, {CVPR} 2018, Salt Lake City, UT, USA, June 18-22, 2018}, 2018,
pp. 8827--8836.
\bibitem{One04}
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying~Nian Wu, and Song{-}Chun Zhu,
\newblock ``Interpreting {CNN} knowledge via an explanatory graph,''
\newblock in {\em Proceedings of the Thirty-Second {AAAI} Conference on
Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February
2-7, 2018}, 2018, pp. 4454--4463.
\bibitem{global03}
Yao Lu,
\newblock ``Unsupervised learning on neural network outputs: With application
in zero-shot learning,''
\newblock in {\em Proceedings of the Twenty-Fifth International Joint
Conference on Artificial Intelligence, {IJCAI} 2016, New York, NY, USA, 9-15
July 2016}, 2016, pp. 3432--3438.
\bibitem{global04}
Mathieu Aubry and Bryan~C. Russell,
\newblock ``Understanding deep features with computer-generated imagery,''
\newblock in {\em 2015 {IEEE} International Conference on Computer Vision,
{ICCV} 2015, Santiago, Chile, December 7-13, 2015}, 2015, pp. 2875--2883.
\bibitem{local01}
Marco~T{\'{u}}lio Ribeiro, Sameer Singh, and Carlos Guestrin,
\newblock ``Why should {I} trust you?: Explaining the predictions of any
classifier,''
\newblock in {\em Proceedings of the 22nd {ACM} {SIGKDD} International
Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA,
August 13-17, 2016}, 2016, pp. 1135--1144.
\bibitem{local02}
Marco~T{\'{u}}lio Ribeiro, Sameer Singh, and Carlos Guestrin,
\newblock ``Model-agnostic interpretability of machine learning,''
\newblock {\em CoRR}, vol. abs/1606.05386, 2016.
\bibitem{local03}
Ruth~C. Fong and Andrea Vedaldi,
\newblock ``Interpretable explanations of black boxes by meaningful
perturbation,''
\newblock in {\em {IEEE} International Conference on Computer Vision, {ICCV}
2017, Venice, Italy, October 22-29, 2017}, 2017, pp. 3449--3457.
\bibitem{local04}
Ramprasaath~R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam,
Devi Parikh, and Dhruv Batra,
\newblock ``Grad-cam: Visual explanations from deep networks via gradient-based
localization,''
\newblock in {\em {IEEE} International Conference on Computer Vision, {ICCV}
2017, Venice, Italy, October 22-29, 2017}, 2017, pp. 618--626.
\bibitem{Inception}
C.~{Szegedy}, {Wei Liu}, {Yangqing Jia}, P.~{Sermanet}, S.~{Reed},
D.~{Anguelov}, D.~{Erhan}, V.~{Vanhoucke}, and A.~{Rabinovich},
\newblock ``Going deeper with convolutions,''
\newblock in {\em 2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, June 2015, pp. 1--9.
\bibitem{ref02}
Trevor~Hastie Jerome~Friedman and Robert Tibshirani,
\newblock ``The elements of statistical learning: Data mining, inference and
prediction,''
\newblock in {\em https://web.stanford.edu/~hastie/ElemStatLearn/}, 2009.
\bibitem{ref01}
Christoph Molnar,
\newblock ``Interpretable machine learning: A guide for making black box models
explainable,''
\newblock in {\em Lulu, 1st edition, March 24, 2019; eBook}, 2019.
\bibitem{ref03}
S.~Ruping,
\newblock ``Learning interpretable models (phd thesis),''
\newblock in {\em Technical University of Dortmund}, 2006.
\end{thebibliography}