FOLD-R++: A Scalable Toolset for Automated Inductive Learning of Default Theories from Mixed Data
Extracted source text · arXiv 2110.07843
% This is samplepaper.tex, a sample chapter demonstrating the
% LLNCS macro package for Springer Computer Science proceedings;
% Version 2.20 of 2017/10/04
%
\documentclass[runningheads]{llncs}
%
\usepackage{graphicx}
\usepackage{adjustbox}
\usepackage{algorithm,algpseudocode}
\newtheorem{exmp}{Example}
\algnewcommand\algorithmicinput{\textbf{Input:}}
\algnewcommand\Input{\item[\algorithmicinput]}
\algnewcommand\algorithmicoutput{\textbf{Output:}}
\algnewcommand\Output{\item[\algorithmicoutput]}
\algnewcommand\algorithmicforeach{\textbf{for each}}
\algnewcommand{\LeftComment}[1]{\Statex \(\triangleright\) #1}
\algdef{S}[FOR]{ForEach}[1]{\algorithmicforeach\ #1\ \algorithmicdo}
\newcommand{\tabitem}{~~\llap{\textbullet}~~}
% Used for displaying a sample figure. If possible, figure files should
% be included in EPS format.
%
% If you use the hyperref package, please uncomment the following line
% to display URLs in blue roman font according to Springer's eBook style:
% \renewcommand\UrlFont{\color{blue}\rmfamily}
\begin{document}
%
\title{FOLD-R++: A Toolset for Automated Inductive Learning of Default Theories from Mixed Data}
%
\titlerunning{FOLD-R++ Toolset}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Huaduo Wang \and
Gopal Gupta }
%
\authorrunning{H. Wang, G. Gupta}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{The University of Texas at Dallas, Richardson TX 75080, USA \\
\email{\{huaduo.wang, gupta\}@utdallas.edu}}
%
\maketitle % typeset the header of the contribution
%
\begin{abstract}
FOLD-R is an automated inductive learning algorithm for learning default rules with exceptions for mixed (numerical and categorical) data. It generates an (explainable) answer set programming (ASP) rule set for classification tasks. We present an improved FOLD-R algorithm, called FOLD-R++, that significantly increases the efficiency and scalability of FOLD-R. FOLD-R++ improves upon FOLD-R without compromising or losing information in the input training data during the encoding or feature selection phase. The FOLD-R++ algorithm is competitive in performance with the widely-used XGBoost algorithm, however, unlike XGBoost, the FOLD-R++ algorithm produces an explainable model. Next, we create a powerful tool-set by combining FOLD-R++ with s(CASP)---a goal-directed ASP execution engine---to make predictions on new data samples using the answer set program generated by FOLD-R++. The s(CASP) system also produces a justification for the prediction. Experiments presented in this paper show that our improved FOLD-R++ algorithm is a significant improvement over the original design and that the s(CASP) system can make predictions in an efficient manner as well.
\keywords{Inductive Logic Programming, Machine Learning, Explainable AI, Negation as Failure, Answer Set Programming, Data mining}
\end{abstract}
%
%
%
\section{Introduction}
Dramatic success of machine learning has led to a torrent of Artificial Intelligence (AI) applications. However, the effectiveness of these systems is limited by the machines' current inability to explain their decisions and actions to human users. That's mainly because the statistical machine learning methods produce models that are complex algebraic solutions to optimization problems such as risk minimization
or geometric margin maximization. Lack of intuitive descriptions makes it hard for users to understand and verify the underlying rules that govern the model. Also, these methods cannot produce a justification for a prediction they arrive at for a new data sample.
The Explainable AI program \cite{xai} aims to create a suite of machine learning techniques that: a) Produce more explainable models, while maintaining a high level of prediction accuracy. b) Enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners. Inductive Logic Programming (ILP) \cite{ilp} is one Machine Learning technique where the learned model is in the form of logic programming rules (Horn Clauses) that are comprehensible to humans. It allows the background knowledge to be incrementally extended without requiring the entire model to be re-learned. Meanwhile, the comprehensibility of symbolic rules makes it easier for users to understand and verify induced models and even refine them.
The ILP learning problem can be regarded as a search problem for a set of clauses that deduce the training examples. The search is performed either top down or bottom-up. A bottom-up approach builds most-specific clauses from the training examples and searches the hypothesis space by using generalization. This approach is not applicable to large-scale datasets, nor it can incorporate \textit{negation-as-failure} into the hypotheses. A survey of bottom-up ILP systems and their shortcomings can be found at \cite{sakama05}. In contrast, top-down approach starts with the most general clause and then specializes it. A top-down algorithm guided by heuristics is better suited for large-scale and/or noisy datasets \cite{quickfoil}.
% hw: introduce both foil and fold, compare them and introduce the fold-r, performance issue on scalability.
The FOIL algorithm \cite{foil} by Quinlan is a popular top-down inductive logic programming algorithm that generate logic programs. FOIL uses weighted information gain as the heuristics to guide the search for best literals. The FOLD algorithm by Shakerin \cite{shakerin-phd,fold} is a new top-down algorithm inspired by the FOIL algorithm. It generalizes the FOIL algorithm by learning default rules with exceptions. It does so by first learning the default conclusion that covers positive examples while avoiding negative examples, then next it swaps the positive and negative examples and calls itself recursively to learn the exceptions to the default conclusions. Both FOIL and FOLD cannot deal with numeric features directly; an encoding process is needed in the preparation phase of the training data that discretizes the continuous numbers into intervals. However, this process not only adds a huge computational overhead to the algorithm but also leads to loss of information in the training data.
To deal with the above problems, Shakerin developed an extension of the FOLD algorithm, called FOLD-R, to handle mixed (i.e., both numerical and categorical) features which avoids the discretization process for numerical data \cite{shakerin-phd,fold}. However, FOLD-R still suffers from efficiency and scalability issues when compared to other popular machine learning systems for classification. In this paper we report on a novel implementation method we have developed to improve the design of the FOLD-R system. In particular, we use the prefix sum technique \cite{prefixsum} to optimize the process of calculation of information gain, the most time consuming component of the FOLD family of algorithms \cite{shakerin-phd}. Our optimization, in fact, reduces the time complexity of the algorithm. If \texttt{N} is the number of unique values from a specific feature and \texttt{M} is the number of training examples, then the complexity of classifying all the examples for each split is reduced from \texttt{O(M*N)} for FOLD-R to \texttt{O(M+N)} in FOLD-R++.
Our experimental results indicate that the FOLD-R++ algorithm is comparable to popular machine learning algorithms such as XGBoost wrt various metrics (accuracy, recall, precision, and F1-score) as well as in efficiency and scalability. However, in addition, FOLD-R++ produces an explainable and interpretable model in the form of an answer set program.
This paper makes the following novel contribution: it presents the FOLD-R++ algorithm that significantly improves the efficiency and scalability of the FOLD-R ILP algorithm without adding overhead during pre-processing or losing information in the training data. As mentioned, the new approach is competitive with popular classification models such as the XGBoost classifier \cite{xgboost}. The FOLD-R++ algorithm outputs an answer set program (ASP) \cite{gelfondkahl} that serves as an explainable/interpretable model. This generated answer set program is compatible with s(CASP) \cite{scasp}, a goal-directed ASP solver, that can efficiently justify the prediction generated by the ASP model.\footnote{The s(CASP) system is freely available at \url{https://gitlab.software.imdea.org/ciao-lang/sCASP}.}
%The rest of the paper is organized as follows.
\section{Background}
\label{sec:background}
% hw: add a passage for inductive learning problem description, rephrase the passage below.
\subsection{Inductive Logic Programming}
Inductive Logic Programming (ILP) \cite{ilp} is a subfield of machine learning that learns models in the form of logic programming rules (Horn Clauses) that are comprehensible to humans. This problem is formally defined as:\\
\textbf{Given}
\begin{enumerate}
\item A background theory $B$, in the form of an extended logic program, i.e., clauses of the form $h \leftarrow l_1, ... , l_m,\ not \ l_{m+1},...,\ not \ l_n$, where $l_1,...,l_n$ are positive literals and \textit{not} denotes \textit{negation-as-failure} (NAF) \cite{Baral,gelfondkahl}. We require that $B$ has no loops through negation, i.e., it is stratified.
\item Two disjoint sets of ground target predicates $E^+, E^-$ known as positive and negative examples, respectively
\item A hypothesis language of function free predicates $L$, and a refinement operator $\rho$ under $\theta-subsumption$ \cite{plotkin70} that would disallow loops over negation.
\end{enumerate}
\textbf{Find} a set of clauses $H$ such that:
\begin{itemize}
\item $ \forall e \in \ E^+ ,\ B \cup H \models e$
\item $ \forall e \in \ E^- ,\ B \cup H \not \models e$
\item $B \land H$ is consistent.
\end{itemize}
\section{The FOLD-R++ Algorithm}
The FOLD algorithm \cite{shakerin-phd,fold} is a top-down ILP algorithm that searches for best literals to add to the body of the clauses for hypothesis, $H$, with the guidance of an information gain-based heuristic. The FOLD-R++ algorithm\footnote{The FOLD-R++ toolset is available on \url{https://github.com/hwd404/FOLD-R-PP}.} refactors the FOLD algorithm and is summarized in algorithm\ref{algo:fold}. The output of the FOLD-R++ algorithm is a set of default rules that include exceptions. An example implied by any rule in the set would be classified as positive. Therefore, the FOLD-R++ algorithm rules out the already covered positive examples in line 5 after learning a new rule. For each rule learning process, a best literal would be selected based on weighted information gain with the current training examples, in line 13, then the examples that cannot be implied by learned default literals would be ruled out for further learning of the current rule. When the information gain becomes zero or the number of negative examples drops below the ratio threshold, the default learning part is done. Unlike the FOIL algorithm, FOLD-R++ next learns exceptions after first learning default literals. This is done by swapping the residual positive and negative examples and calling itself recursively in line 29. The remaining positive and negative examples can be swapped again and exceptions to exceptions learned (and then swapped further to learn exceptions to exceptions of exceptions, and so on). The $ratio$ parameter in Algorithm \ref{algo:fold} represents the ratio of training examples that are part of the exception to the examples implied by only the default conclusion part of the rule. It will allow us to control the nesting level of exceptions the user wants to permit.
\begin{algorithm}[!h]
\caption{FOLD-R++ Algorithm}
\label{algo:fold}
\begin{algorithmic}[1]
\Input $target,B,E^+,E^-,ratio$ \Comment{$ratio$ is the exception ratio}
\Output $R = \{r_1,...,r_n\}$ \Comment{R is rule set}
% \Statex $AB = \{ab_1,...,ab_m\}$ \Comment{exceptions/abnormal clauses}
\Function{FOLD-R++}{$E^+,E^-,L_{used}$}
\State $R \gets \emptyset$
\While{$|E^+| > 0$}
\State $r \gets$ \Call{learn\_rule}{{$E^+$},{$E^-$},{$L_{used}$}}
\State $E^+ \gets E^+ \setminus covers(r,E^+,true)$
\State $R \gets R \cup \{ r \}$
\EndWhile
\State \Return $R$
\EndFunction
\Function{LEARN\_RULE}{${E^+},{E^-},{L_{used}}$}
%\State $first\_iteration \gets true$
%\While{($size(E^-) > 0 $)}
% \While{$|E^-|>0 \land c.length < max\_rule\_length$}
\State $L \gets \emptyset$
\While{$true$}
\State $l \gets$ \Call{find\_best\_literal}{{$E^+$},{{$E^-$}},{$L_{used}$}}
\State $L \gets L \cup \{ l \}$
\State $r \gets set\_default(r,L)$
\State $E^+ \gets covers(r,E^+,true)$
\State $E^- \gets E^- \setminus covers(r,E^-,false)$
\If{$l$ is invalid or $|E^-| \leq |E^+| * ratio$}
\If{$l$ is invalid}
\State $L \gets L \setminus \{ l \}$
\State $r \gets set\_default(r,L)$
\Else
\State $flag \gets true$
\EndIf
\State break
\EndIf
\EndWhile
\If{$flag$}
\State $AB \gets$ \Call{fold-r++}{{$E^-$},{{$E^+$}},{$L_{used} + L$}}
\State $r \gets set\_exception(r,AB)$
\EndIf
\State \Return $r$
\EndFunction
\end{algorithmic}
\end{algorithm}
\begin{exmp}
\label{ex:pinguin}
In the FOLD-R++ algorithm, the target is to learn rules for \texttt{fly(X)}. $ B, E^+, E^-$ are background knowledge, positive and negative examples, respectively.
\end{exmp}
\begin{verbatim}
B: bird(X) :- penguin(X).
bird(tweety). bird(et).
cat(kitty). penguin(polly).
E+: fly(tweety). fly(et).
E-: fly(kitty). fly(polly).
\end{verbatim}
The target predicate \texttt{\{fly(X) :- true.\}} is specified when calling the learn\_rule function at line 4. The function selects the literal \texttt{bird(X)} as result and adds it to the clause $r$ = \texttt{fly(X) :- bird(X)} because it has the best information gain among \texttt{\{bird,penguin,cat\}}. Then, the training set gets updated to $E^+=\{tweety, et\}$,\ $E^-=\{polly\}$ in line 15-16. The negative example $polly$ is still implied by the generated clause and so is a false negative classification. The default learning of learn\_rule function is finished because the best information gain of candidate literal is zero. Therefore, the FOLD-R++ function is called recursively with swapped positive and negative examples, $E^+ = \{polly\}$, $E^-=\{tweety, et\}$, to learn exceptions. In this case, an abnormal predicate \texttt{\{ab0(X) :- penguin(X)\}} is generated and returned as the only exception to the previous learned clause as $r$ = \texttt{fly(X) :- bird(X), ab0(X)}. The abnormal rule \texttt{\{ab0(X) :- penguin(X)\}} is added to the final rule set producing the program below:
\begin{center}
\begin{tabular}{l}
\texttt{fly(X) :- bird(X), not ab0(X).}\\
\texttt{ab0(X) :- penguin(X).}
\end{tabular}
\end{center}
% hw: mention that most time consuming function by the algorithm on a certain level would be the select literal function. describe the discretization on numeric features and the original design of the numeric extension. say sth. about information loss and then give the new optimization, which includes the formula of information gain and the loop of calculation process.
% hw: based on the pseudo code above, provide an example of the application of prefix sum on infogain calculation. and analyze the time complexity for ig.
% hw: compare the execution time with original paper on several datasets, credit-au and mushroom, ecoli.
% hw: compare with new implementation of
\noindent We next give more details of the FOLD-R++ algorithm.
\subsection{Literal Selection}
The literal selection process for Shakerin's FOLD-R algorithm can be summarized in Algorithm \ref{algo:fold-r-specialize}.
\begin{algorithm}[!h]
\caption{FOLD-R Algorithm's Specialize function}
\label{algo:fold-r-specialize}
\begin{algorithmic}[1]
% \Input $E^+,E^-,i,L_{used}$
% \Output $best, l$ \Comment{}
\Function{SPECIALIZE}{$c,E^+,E^-$}
\While{$size(E^-)>0$}
\State $(c_1,IG_1) \gets test\_categorical(c,E^+,E^-)$
\State $(c_2,IG_2) \gets test\_numeric(c,E^+,E^-)$
\If{$IG_1=0$ \& $IG_2=0$}
\State $\hat{c} \gets EXCEPTION(c,E^-,E^+)$
\If{$\hat{c}=null$}
\State $\hat{c} \gets enumerate(c,E^+)$
\EndIf
\Else
\If{$IG_1 \geq IG_2$}
\State $\hat{c} \gets c_1$
\Else
\State $\hat{c} \gets c_2$
\EndIf
\EndIf
\State $E^- \gets E^- \setminus covers(\hat{c}, E^-)$
\EndWhile
\EndFunction
\end{algorithmic}
\end{algorithm}
The FOLD-R algoirthm \cite{shakerin-phd,fold} selects the best literal based on the weighted information gain for learning defaults, similar to the original FOLD algorithm described in \cite{fold}. For numeric features, the FOLD-R algorithm would enumerate all the possible splits. Then, it classifies the data and compute information gain for literals for each split. The literal with the best information gain would be selected as result. In contrast, FOLD-R++ algorithm uses a new, more efficient method employing \textit{prefix sums} to calculate the information gain based on the classification categories. In FOLD-R++, information gain for a given literal is calculated as shown in Algorithm \ref{algo:fold-r-plus-ig}.
\begin{algorithm}[!h]
\caption{FOLD-R++ Algorithm, Information Gain function}
\label{algo:fold-r-plus-ig}
\begin{algorithmic}[1]
% \Input $E^+,E^-,i,L_{used}$
% \Output $best, l$ \Comment{}
\Function{IG}{$tp,fn,tn,fp$}
\If{$fp+fn>tp+tn$}
\State \Return $-\infty$
\EndIf
\State $pos, neg \gets tp+fp, tn+fn$
\State $tot \gets pos+neg$
\State $result \gets (\frac{tp}{tot} log_2(\frac{tp}{pos}))_{tp>0} + (\frac{fp}{tot} log_2(\frac{fp}{pos}))_{fp>0}$
\State $result \gets result + (\frac{tn}{tot} log_2(\frac{tn}{neg}))_{tn>0} + (\frac{fn}{tot} log_2(\frac{fn}{neg}))_{fn>0}$
\State \Return $result$
\EndFunction
\end{algorithmic}
\end{algorithm}
% \begin{equation}
% IG(tp, fn, tn, fp) = \begin{cases}
% $-\infty$ &\text{$fp+fn>tp+tn$}\\
% \frac{1}{tp+fn+tn+fp} \left( F(tp, fp) + F(tn, fn) \right)
% \end{cases}
% \end{equation}
% \begin{equation}
% F(x, y) = x * log_2(\frac{x}{x+y}) + y * log_2(\frac{y}{x+y})
% \end{equation}
\noindent The variables $tp, fn, tn, fp$ in Algorithm \ref{algo:fold-r-plus-ig} for finding the information gain represent the numbers of true positive, false positive, true negative, and false negative examples, respectively. With the function above, the new approach employs the \textit{prefix sum technique} to speed up the calculation. Only one round of classification is needed for a single feature, even with mixed types of values. The new approach to calculate the best IG and literal is summarized in Algorithm \ref{algo:best_ig}.
\begin{algorithm}[!h]
\caption{FOLD-R++ Algorithm, Best Information Gain function}
\label{algo:best_ig}
\begin{algorithmic}[1]
\Input $E^+,E^-,i$ %,L_{used}$
\Output $best, l$ \Comment{$best$: the best IG of feature $i$, $l$: the literal with IG $best$}
\Function{Best\_info\_gain}{$E^+,E^-,i$}%,L_{used}$}
\State $pos, neg \gets count\_classification(E^+, E^-, i)$
\LeftComment{pos, neg are dicts that holds the \# of pos / neg examples for each value}
\State $xs, cs \gets collect\_unique\_values(E^+, E^-, i)$
\LeftComment{xs, cs are lists that holds the unique numeric and categorical values}
\State $xp, xn, cp, cn \gets count\_total(E+, E-, i)$
\LeftComment{(xp, xn) are the total \# of pos / neg examples with numeric value, (cp, cn) are the same for categorical values.}
\State $xs \gets sort(xs)$
\For{$j \gets 1$ to $size(xs)$} \Comment{compute the prefix sum}
\State $pos[xs_{i}] \gets pos[xs_{i}] + pos[xs_{i-1}]$
\State $neg[xs_{i}] \gets neg[xs_{i}] + neg[xs_{i-1}]$
\EndFor
\For{$x \in xs$}
% \If{$literal(i,\leq, x) \notin L_{used}$ \& $literal(i,>, x) \notin L_{used}$}
\State $lit\_dict[literal(i,\leq, x)] \gets IG(pos[x], xp-pos[x]+cp,xn-neg[x]+cn,neg[x])$
\State $lit\_dict[literal(i,>, x)] \gets IG(xp-pos[x],pos[x]+cp,neg[x]+cn,xn-neg[x])$
% \EndIf
\EndFor
\For{$c \in cs$}
% \If{$literal(i,=, x) \notin L_{used}$ \& $literal(i,\ne, x) \notin L_{used}$}
\State $lit\_dict[literal(i,=, x)] \gets IG(pos[c],cp-pos[c]+xp,cn-neg[c]+xn,neg[c])$
\State $lit\_dict[literal(i,\ne, x)] \gets IG(cp-pos[c]+xp,pos[c],neg[c],cn-neg[c]+xn)$
% \EndIf
\EndFor
\State $best, l \gets best\_pair(lit\_dict)$
\State \Return $best, l$
\EndFunction
\end{algorithmic}
\end{algorithm}
\begin{exmp}
\label{ex:pinguin2}
Given positive and negative examples, $E^+, E^-$, with mixed type of values on feature $i$, the target is to find the literal with the best information gain on the given feature. There are $8$ positive examples, their values on feature $i$ are $[1,2,3,3,5,6,6,b]$. And, the values on feature $i$ of the $5$ negative examples are $[2,4,6,7,a]$.
\end{exmp}
\noindent
With the given examples and specified feature, the numbers of positive examples and negative examples for each unique value are counted first, which are shown as $pos, neg$ at right side of Table \ref{tbl:example1}. Then, the prefix sum arrays are calculated for computing heuristic as $pos\_sum, neg\_sum$. Table \ref{tbl:example2} show the information gain for each literal, the $literal(i, \ne, a)$ has been selected with the highest score.
\begin{table}[]
\centering
\begin{tabular}{|c|l|}
\hline
\multicolumn{1}{|l|}{\textbf{examples}} & \textbf{ith feature value} \\ \hline
$\mathbf{E^+}$ & 1 2 3 3 5 6 6 b \\ \hline
$\mathbf{E^-}$ & 2 4 6 7 a \\ \hline
\end{tabular}
\quad
% \begin{tabular}{|l|c|c|c|c|c|c|c|c|c|}
\begin{tabular}{|l|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|}
\hline
\textbf{value} & 1 & 2 & 3 & 4 & 5 & 6 & 7 & a & b \\ \hline
\textbf{pos} & 1 & 1 & 2 & 0 & 1 & 2 & 0 & 0 & 1 \\ \hline
\textbf{pos\_sum} & 1 & 2 & 4 & 4 & 5 & 7 & 7 & na & na \\ \hline
\textbf{neg} & 0 & 1 & 0 & 1 & 0 & 1 & 1 & 1 & 0 \\ \hline
\textbf{neg\_sum} & 0 & 1 & 1 & 2 & 2 & 3 & 4 & na & na \\ \hline
\end{tabular}
\caption{Left: Examples and values on i$^{th}$ feature. Right: positive/negative count and prefix sum on each value }
\label{tbl:example1}
\end{table}
\begin{table}[]
\centering
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|}
\hline
& \multicolumn{9}{c|}{Info Gain} \\ \hline
\textbf{value} & 1 & 2 & 3 & 4 & 5 & 6 & 7 & a & b \\ \hline
\textbf{$\leq$ value} & $-\infty$ & $-\infty$ & -0.619 & -0.661 & -0.642 & -0.616 & -0.661 & na & na \\ \hline
\textbf{$>$ value} & -0.664 & -0.666 & $-\infty$ & $-\infty$ & $-\infty$ & $-\infty$ & $-\infty$ & na & na \\ \hline
\textbf{$=$ value} & na & na & na & na & na & na & na & $-\infty$ & $-\infty$ \\ \hline
\textbf{$\ne$ value} & na & na & na & na & na & na & na & \textbf{-0.588} & -0.627 \\ \hline
\end{tabular}
\caption{The info gain on i$^{th}$ feature with given examples}
\label{tbl:example2}
\end{table}
\subsection{Justification}
Explainability is very important for some tasks like loan approval, credit card approval, and disease diagnosis system. Answer set programming provides explicit rules for how a prediction is generated compared to black box models like those based on neural networks. To efficiently justify the prediction, the FOLD-R++ outputs answer set programs that are compatible with the s(CASP) goal-directed ASP system \cite{scasp}.
\begin{exmp}
\label{ex:car}
The ``Titanic Survival Prediction" is a classical classification challenge which contains 891 passengers as training examples and 418 passengers as testing examples and their survival based on features such as sex, age, number of siblings/spouses, number of parents/children, etc.. FOLD-R++ generates the following program with only 12 rules!:
\end{exmp}
\iffalse
{\scriptsize
\begin{verbatim}
(1) ab1(X):- fare(X,N4), N4=<28.712, fare(X,N4), N4>27.75.
(2) ab2(X):- age(X,N1), N1>16.0, fare(X,N4), N4=<7.75, age(X,N1), N1=<21.0.
(3) ab3(X):- sex(X,'female'), fare(X,N4), N4=<10.463.
(4) ab4(X):- fare(X,N4), N4>81.858, sex(X,'female').
(5) survived(X,'1'):- age(X,N1), N1=<6.0, number_of_siblings_spouses(X,N2), N2=<2.0,
not ab3(X), not ab4(X).
(6) survived(X,'1'):- sex(X,'female'), fare(X,N4), N4=<23.25, not embarked(X,'s'),
fare(X,N4), N4=<15.5, fare(X,N4), N4=<13.417, not ab2(X).
(7) survived(X,'1'):- sex(X,'female'), not class(X,'3'), not ab1(X).
\end{verbatim}
}
\fi
{\scriptsize
\begin{verbatim}
(1) status(X,0):- sex(X,'male'), not ab1(X), not ab3(X), not ab5(X).
(2) status(X,0):- class(X,'3'), not sex(X,'male'), fare(X,N4), N4>23.25,
not ab6(X), not ab7(X).
(3) status(X,0):- class(X,'3'), not sex(X,'male'), age(X,N1),N1>16.0,
number_of_siblings_spouses(X,N2), N2=<2.0, fare(X,N4), N4>7.65,
number_of_parents_children(X,N3), N3=<1.0, fare(X,N4), N4=<18.0,
fare(X,N4), N4>12.475, not ab8(X), not ab9(X).
(4) ab1(X):- fare(X,N4), N4>26.25, number_of_siblings_spouses(X,N2), N2>2.0, age(X,N1),
N1=<3.0, age(X,N1), N1>2.0.
(5) ab2(X):- age(X,N1), N1>42.0, age(X,N1), N1=<45.0, number_of_siblings_spouses(X,N2),
N2=<0.0, number_of_parents_children(X,N3), N3=<0.0, embarked(X,'s').
(6) ab3(X):- class(X,'1'), age(X,N1), N1=<52.0, fare(X,N4), N4>25.587, fare(X,N4),
N4=<26.55, not ab2(X).
(7) ab4(X):- number_of_parents_children(X,N3), N3=<0.0, age(X,N1), N1=<11.0.
(8) ab5(X):- age(X,N1), N1=<12.0, number_of_siblings_spouses(X,N2), N2=<2.0, not ab4(X).
(9) ab6(X):- number_of_parents_children(X,N3), N3=<0.0.
(10) ab7(X):- fare(X,N4), N4>31.275, fare(X,N4), N4=<31.387.
(11) ab8(X):- fare(X,N4), N4>15.5, fare(X,N4), N4=<17.4, age(X,N1), N1=<24.0.
(12) ab9(X):- age(X,N1), N1>32.0, age(X,N1), N1=<36.0.
\end{verbatim}
}
\noindent
Note that {\tt status(X,0)} means that person whose id is {\tt X} perished, while {\tt status(X,1)} means that person with id {\tt X} survived. Note that we don't have any rules generated for {\tt status(X,1)}, so we could add a rule: {\tt status(X,1) :- not status(X,0).} The above program achieves 0.94 accuracy, 0.97 precision, 0.93 recall, and 0.95 $F_1$ score, which is quite remarkable. Given a new data sample, the predicted answer for this data sample using the above answer set program can be efficiently produced by the s(CASP) system. The s(CASP) system can also produce a justification (a proof tree) for this prediction. Since s(CASP) is query driven, an example query such as {\tt ?- status(926, S)} which checks if passenger with id 926 perished or survived, will succeed if status of passenger 926 is indeed predicted as perished ({\tt S} is set to 0) by the model represented by the answer set program above. The s(CASP) system can provide a proof for each query. The English description for predicates is also needed to output the proof tree in human readable format. The meaning of predicates in English is given via the {\tt \#pred} declaration, as shown below via examples:
{\scriptsize
\begin{verbatim}
#pred age(X,Y) :: 'person @(X) is of age @(Y)'.
#pred number_of_sibling_spouses(X,Y) :: 'person @(X) had @(Y) siblings or spouses'.
#pred ab9(X) :: 'abnormal case 9 holds for @(X)'.
\end{verbatim}
}
\noindent The s(CASP) system can even generate this proof in a human understandable form \cite{arias-justification}. For example, here is the justification tree generated for the passenger with id 926:
{\scriptsize
\begin{verbatim}
?- status(926,X).
% QUERY:I would like to know if
'status' holds (for 926, and X).
ANSWER: 1 (in 4.825 ms)
JUSTIFICATION_TREE:
person 926 perished, because
person 926 is male, and
there is no evidence that 'ab1' holds (for 926), because
there is no evidence that person 926 paid Var1 not equal 57.75 for the ticket, and
person 926 paid 57.75 for the ticket, and
there is no evidence that 'number_of_siblings_spouses' holds (for 926, and Var8).
there is no evidence that abnormal case 3 holds for 926, because
there is no evidence that 'class' holds (for 926, and 1).
there is no evidence that abnormal case 5 holds for 926, because
there is no evidence that person 926 is of age Var2 not equal 30, and
person 926 is of age 30.
The global constraints hold.
\end{verbatim}
}
With the justification tree, the reason for the prediction can be easily understood by human beings. The generated ASP rule-set can also be understood by a human. In fact, s(CASP) can print the ASP rules in English, given the description of predicates in English via the {\tt \#pred} declaration explained above. If there is any unreasonable logic generated in the rule set, it can also be modified directly by the human without retraining. Thus, any bias in the data that is captured in the generated ASP rules can be corrected by the human user, and the updated ASP rule-set used for making new predictions. An example translation for two of the rules (Rules (1) and (12)) above is shown below:
{\scriptsize
\begin{verbatim}
(1) person X perished, if
person X is male and
there is no evidence that 'ab1' holds (for X) and
there is no evidence that abnormal case 3 holds for X and
there is no evidence that abnormal case 5 holds for X.
(12) abnormal case 9 holds for X, if
person X is of age Y and
Y is greater than 32.0 and
person X is of age Y and
Y is less or equal 36.0.
\end{verbatim}
}
Note that if a data sample is not predicted to hold, because the corresponding query fails on s(CASP), then a justification can be generated by asking the negation of the query. The s(CASP) system supports constructive negation, and thus negated queries can be executed in s(CASP) and their justification/proof generated just as easily as the positive queries.
\section{Experiments}
\label{sec:experiments}
% \footnote{Full implementation is available at: \url{https://github.com/fxs130430/SHAP{\_}FOLD}}
In this section, we present our experiments on UCI standard benchmarks \cite{uci}. The XGBoost Classifier is popular classification model and used as a baseline in our experiment. We used simple settings for XGBoost classifier without limiting its performance. However, XGBoost cannot deal with mixed type (numerical and categorical) of examples directly. One-hot encoding has been used for data preparation. We use precision, recall, accuracy, $F_1$ score, and execution time to compare the results.
FOLD-R++ does not requires any encoding before training. The original FOLD-R system used the JPL library with Java implementation. We implemented FOLD-R++ only with Python. To make inferences using the generated rules, we developed a simple ASP interpreter for our application that is part of the FOLD-R++ system. Note that the generated programs are stratified and predicates contain only variables and constants, so implementing an interpreter for such a restricted class in Python is relatively easy. However, for obtaining the justification/proof tree, or for translating the ASP rules into equivalent English text, one must use the s(CASP) system.
As discussed earlier, the time complexity for computing information gain on a feature is significantly reduced in FOLD-R++ due to the use of prefix-sum. Therefore, we obtain a rather large improvements in efficiency. For the credit-a dataset with only 690 instances, the new FOLD-R++ algorithm is hundreds times faster than the original FOLD-R. All the learning experiments have been conducted on a desktop with Intel i5-10400 CPU @ 2.9GHz and 32 GB ram. To measure performance metrics, we conducted 10-fold cross-validation on each dataset and the average of accuracy, precision, recall, $F_1$ score and execution time have been presented. Table \ref{tbl:accuracies} reports the performance metrics and execution time on each dataset compared with the baseline model. The best performer is highlighted with boldface font.
\begin{table}[]
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|c|}
\hline
& \multicolumn{1}{l|}{} & \multicolumn{10}{c|}{Algorithm} \\ \hline
& \multicolumn{1}{l|}{} & \multicolumn{5}{c|}{XGBoost.Classifier} & \multicolumn{5}{c|}{FOLD-R++} \\ \hline
Data Set & Shape & Accuracy & Precision & Recall & F1 & \multicolumn{1}{l|}{Time (ms)} & Accuracy & Precision & Recall & F1 & \multicolumn{1}{l|}{Time (ms)} \\ \hline
acute & (120, 7) & \textbf{1} & 1 & \textbf{1} & \textbf{1} & 35 & 0.99 & 1 & 0.99 & 0.99 & \textbf{2.5} \\ \hline
autism & (704, 18) & \textbf{0.97} & \textbf{0.98} & \textbf{0.98} & 0.97 & 76 & 0.95 & 0.96 & 0.97 & 0.97 & \textbf{47} \\ \hline
breast-w & (699, 10) & 0.95 & 0.97 & 0.96 & 0.96 & 78 & \textbf{0.96} & 0.97 & 0.96 & \textbf{0.97} & \textbf{28} \\ \hline
cars & (1728, 7) & \textbf{1} & 1 & \textbf{1} & \textbf{1} & 77 & 0.98 & 1 & 0.97 & 0.98 & \textbf{48} \\ \hline
credit-a & (690, 16) & \textbf{0.85} & 0.83 & \textbf{0.83} & 0.83 & 368 & 0.84 & \textbf{0.92} & 0.79 & \textbf{0.84} & \textbf{100} \\ \hline
ecoli & (336, 9) & 0.76 & 0.76 & 0.62 & 0.68 & 165 & \textbf{0.96} & \textbf{0.95} & \textbf{0.94} & \textbf{0.95} & \textbf{28} \\ \hline
heart & (270, 14) & \textbf{0.80} & \textbf{0.81} & 0.83 & 0.81 & 112 & 0.79 & 0.79 & 0.83 & 0.81 & \textbf{44} \\ \hline
ionosphere & (351, 35) & 0.88 & 0.86 & \textbf{0.96} & 0.90 & 1,126 & \textbf{0.92} & \textbf{0.93} & 0.94 & \textbf{0.93} & \textbf{392} \\ \hline
kidney & (400, 25) & 0.98 & 0.98 & 0.98 & 0.98 & 126 & \textbf{0.99} & \textbf{1} & 0.98 & \textbf{0.99} & \textbf{27} \\ \hline
kr vs. kp & (3196, 37) & 0.99 & 0.99 & 0.99 & 0.99 & \textbf{210} & 0.99 & 0.99 & 0.99 & 0.99 & 361 \\ \hline
mushroom & (8124, 23) & 1 & 1 & 1 & 1 & \textbf{378} & 1 & 1 & 1 & 1 & 476 \\ \hline
sonar & (208, 61) & 0.53 & 0.54 & \textbf{0.84} & 0.65 & 1,178 & \textbf{0.78} & \textbf{0.81} & 0.75 & \textbf{0.78} & \textbf{419} \\ \hline
voting & (435, 17) & 0.95 & 0.94 & \textbf{0.95} & 0.94 & 49 & 0.95 & 0.94 & 0.94 & 0.94 & \textbf{16} \\ \hline
adult & (32561, 15) & \textbf{0.86} & \textbf{0.88} & 0.94 & \textbf{0.91} & 274,655 & 0.84 & 0.86 & \textbf{0.95} & 0.90 & \textbf{10,069} \\ \hline
credit card & (30000, 24) & - & - & - & - & - & \textbf{0.82} & \textbf{0.83} & \textbf{0.96} & \textbf{0.89} & \textbf{21,349} \\ \hline
\end{tabular}
\end{adjustbox}
\caption{Evaluation of FOLD-R++ on UCI Datasets}
\label{tbl:accuracies}
\end{table}
\iffalse
\begin{table}[]
\begin{adjustbox}{max width=\textwidth}
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|c|}
\hline
& \multicolumn{1}{l|}{} & \multicolumn{10}{c|}{Algorithm} \\ \hline
& \multicolumn{1}{l|}{} & \multicolumn{5}{c|}{Aleph} & \multicolumn{5}{c|}{SHAP-FOLD} \\ \hline
Data Set & Shape & Precision & Recall & Accuracy & F1 & \multicolumn{1}{l|}{Time (s)} & Precision & Recall & Accuracy & F1 & \multicolumn{1}{l|}{Time (s)} \\ \hline
cars & (1728, 6) & 0.83 & 0.63 & 0.85 & 0.72 & 73 & \textbf{0.84} & \textbf{0.94} & \textbf{0.93} & \textbf{0.89} & \textbf{5} \\ \hline
credit-a & (690, 15) & 0.78 & 0.72 & 0.78 & 0.75 & 180 & \textbf{0.90} & \textbf{0.74} & \textbf{0.84} &\textbf{ 0.81} & \textbf{7} \\ \hline
breast-w & (699, 9) & 0.92 & 0.87 & 0.93 & 0.89 & 10 & \textbf{0.92} & \textbf{0.95} & \textbf{0.95} &\textbf{ 0.93} & \textbf{2} \\ \hline
kidney & (400, 24) & \textbf{0.96} & 0.92 & 0.93 & 0.94 & 5 & 0.93 & \textbf{0.95} & 0.93 & 0.94 & \textbf{1} \\ \hline
voting & (435, 16) & 0.97 & 0.94 & 0.95 & 0.95 & 25 & \textbf{0.98} & \textbf{0.98} & 0.95 & \textbf{0.96} & \textbf{1 } \\ \hline
autism & (704, 17) & 0.73 & 0.43 & 0.79 & 0.53 & 476 & \textbf{0.96} & \textbf{0.83} & \textbf{0.95} & \textbf{0.89} & \textbf{2} \\ \hline
ionosphere & (351, 34) & \textbf{0.89} & 0.87 & 0.85 & 0.88 & 113 & 0.87 & \textbf{0.91} & 0.85 & \textbf{0.89} & \textbf{2} \\ \hline
heart & (270, 13) & 0.76 & 0.75 & 0.78 & 0.75 & 28 & 0.76 & \textbf{0.83} & \textbf{0.81} & \textbf{0.80} & \textbf{1} \\ \hline
kr vs. kp & (3196, 36) & 0.92 & 0.99 & 0.95 & 0.95 & 836 & 0.92 & 0.99 & 0.95 & 0.95 & \textbf{8} \\ \hline
\end{tabular}
\end{adjustbox}
\caption{Evaluation of SHAP\_FOLD on UCI Datasets}
\label{tbl:accuracies}
\end{table}
\fi
The XGBoost Classifier employs decision tree ensemble method for classification task and provides quite decent performance. FOLD-R++ almost always spends less time to finish learning compared to XGBoost classifier, especially for the large dataset Adult income census. For most of the datasets, FOLD-R++ can achieve equivalent scores. FOLD-R++ achieves much higher scores on ecoli and sonar datasets. For the credit card dataset, the baseline XGBoost model failed training due to 32 GB memory limitation, but FOLD-R++ still finished training quite efficiently.
\section{Related Works}
\label{sec:relatedworks}
%\section{Conclusions}
%\label{sec:conclusion}
ALEPH \cite{aleph} is one of the most popular ILP system, which induces theories by using bottom-up generalization search. However, it cannot deal with numeric features and its specialization step is manual, there is no automation option.
Takemura and Inoue's method \cite{Akihiro2021} relies on tree-ensembles to generate explainable rule sets with pattern mining techniques. Its performance depends on the tree-ensemble model. Additionally, it may not be scalable due to its computational time complexity that is exponential in the number of valid rules.
A survey of ILP can be found in \cite{ilp20}. Rule extraction from statistical Machine Learning models has been a long-standing goal of the community. The rule extraction algorithms from machine learning models are classified into two categories: 1) Pedagogical (i.e., learning symbolic rules from black-box classifiers without opening them) 2) Decompositional (i.e., to open the classifier and look into the internals). TREPAN \cite{trepan} is a successful pedagogical algorithm that learns decision trees from neural networks. SVM+Prototypes \cite{svmplus} is a decompositional rule extraction algorithm that makes use of KMeans clustering to extract rules from SVM classifiers by focusing on support vectors. Another rule extraction technique that is gaining attention recently is ``RuleFit" \cite{rulefit}. RuleFit learns a set of weighted rules from ensemble of shallow decision trees combined with original features. In ILP community also, researchers have tried to combine statistical methods with ILP techniques. Support Vector ILP \cite{svmilp} uses ILP hypotheses as kernel in dual form of the SVM algorithm. kFOIL \cite{kfoil} learns an incremental kernel for SVM algorithm using a FOIL style specialization. nFOIL \cite{nfoil} integrates the Naive-Bayes algorithm with FOIL. The advantage of our research over all of the above mentioned research work is that we generate answer set programs containing negation-as-failure that correspond closely to the human thought process. Thus, the descriptions are more concise. Second it is scalable thanks to the greedy nature of our clause search.
\section{Conclusions}
\label{sec:conclusions}
In this paper we presented an efficient and highly scalable algorithm, FOLD-R++, to induce default theories represented as an answer set program. The resulting answer set program has good performance wrt prediction and justification for the predicted classification. In this new approach, unlike other methods, the encoding for data is not needed anymore and no information from training data is discarded. Compared with the popular classification system XGBoost, our new approach has similar performance in terms of accuracy, precision, recall, and F1-score, but better training efficiency. In addition, the FOLD-R++ algorithm produces an explainable model. Predictions made by this model can be computed efficiently and their justification automatically produced using the s(CASP) system.
\iffalse
There are number of directions for future work: (i) Incorporating Relational features, and learning probabilistic logic programs that are currently absent from the research presented in this paper (ii) extending non-monotonic logic programs to image classification explanation domain.
\fi
\section*{Acknowledgement}
Authors gratefully acknowledge support from NSF grants IIS 1718945, IIS 1910131, IIP 1916206, and from Amazon Corp, Atos Corp and US DoD. Views expressed are authors' own and not of the funding agencies.
% BibTeX users please use one of
\bibliographystyle{splncs04}
\bibliography{myilp} % name your BibTeX data base
\end{document}
\iffalse
\begin{algorithm}[!h]
\caption{FOLD Algorithm}
\label{algo:fold}
\begin{algorithmic}[1]
\Input $target,B,E^+,E^-$
\Output $R = \{r_1,...,r_n\}$ \Comment{rule set}
% \Statex $AB = \{ab_1,...,ab_m\}$ \Comment{exceptions/abnormal clauses}
\Function{FOLD}{$E^+,E^-,L_{used}$}
\State $R \gets \emptyset$
\While{$|E^+| > 0$}
\State $r \gets$ \Call{learn\_rule}{{$E^+$},{$E^-$},{$L_{used}$}}
\State $E^+ \gets E^+ \setminus covers(r,E^+,True)$
\State $R \gets R \cup \{ r \}$
\EndWhile
\State \Return $R$
\EndFunction
\Function{LEARN\_RULE}{${E^+},{E^-},{L_{used}}$}
%\State $first\_iteration \gets true$
%\While{($size(E^-) > 0 $)}
% \While{$|E^-|>0 \land c.length < max\_rule\_length$}
\State $L \gets \emptyset$
\While{$true$}
\State $l \gets$ \Call{add\_best\_literal}{{$E^+$},{{$E^-$}},{$L_{used}$}}
\State $L \gets L \cup \{ l \}$
\State $r \gets set\_default(r,L)$
\State $E^+ \gets covers(r,E^+,true)$
\State $E^- \gets E^- \setminus covers(r,E^-,false)$
\If{$l$ is invalid or $|E^-| == 0$}
\If{$l$ is invalid}
\State $L \gets L \setminus \{ l \}$
\State $r \gets set\_default(r,L)$
\Else
\State $flag \gets true$
\EndIf
\State break
\EndIf
\EndWhile
\If{$flag$}
\State $AB \gets$ \Call{fold}{{$E^-$},{{$E^+$}},{$L_{used} + L$}}
\State $r \gets set\_exception(r,AB)$
\EndIf
\State \Return $r$
\EndFunction
\end{algorithmic}
\end{algorithm}
\fi
\iffalse
\begin{algorithm}
\caption{Summarizing the FOIL algorithm}
\label{algo:foil}
\begin{algorithmic}[1]
\Input $target,B,E^+,E^-$
\Output
Initialize $H \gets \emptyset $
\While{($|E^+| > 0$)}
\State $c \gets (target$ :- $ \ true.)$
\While{($|E^-| > 0 \land c.length < max\_length $)}
\For{all $ \ c' \in \rho (c)$}
\State $compute \ score(E^+,E^-,H \cup \{c'\},B)$
\EndFor
\State let $\hat{c}$ be the $c' \in \rho(c)$ with the best score
\State $E^- \gets covers(\hat{c},E^-)$
\EndWhile
\State add $\hat{c}$ to $H$
\State $E^+ \gets E^+ \setminus covers(\hat{c},E^+)$
\EndWhile
\State \textbf{return} $H$
\end{algorithmic}
\end{algorithm}
\fi
\iffalse
The FOIL algorithm by Quinlan \cite{foil} is a popular ILP algorithm that incorporates heuristics from information theory called \textit{weighted information gain} to guide the search for best clauses. The use of a greedy heuristic makes FOIL fast and scalable. However, scalability comes at the expense of losing accuracy if the algorithm is stuck in a local optima and/or when the number of examples is insufficient. Figure \ref{fig:localoptima} demonstrates how the local optima results in discovering sub-optimal rules that does not necessarily coincide with the real underlying sub-concepts of the data.
\begin{figure}
\centering
\includegraphics[width=0.7\textwidth,scale=0.2]{local_optima_issue.png}
\caption{Optimal sequential covering with 3 Clauses (Left), Sub-Optimal sequential covering with 4 Clauses (Right)}
\label{fig:localoptima}
\end{figure}
Unlike FOIL, statistical machine learning algorithms are bound to find the relevant features because they optimize an objective function with respect to global constraints. This results in models that are inherently complex and cannot explain what features account for a classification decision on any given data sample. The Explainable AI techniques such as LIME \cite{lime} and SHAP \cite{shap} have been proposed that provide explanations for any given data sample. Each explanation is a set of feature-value pairs that would locally determine what features and how strongly each feature, relative to other features, contributes to the classification decision. To capture the global behavior of a black-box model, however, an algorithm needs to group similar data samples (i.e., data samples for which the same set of feature values are responsible for the choice of classification) and cover them with the same clause. While in FOIL, the search for a clause is guided by heuristics, in our novel approach, we adapt \textit{High Utility Item-set Mining} (HUIM) \cite{huim}---a popular technique from data mining---to find clauses. We call this algorithm SHAP-FOLD from here on. The advantage of SHAP-FOLD over heuristics-based algorithms such as FOIL is that:
\begin{enumerate}
\item SHAP-FOLD does not get stuck in a local optima
\item SHAP-FOLD distinguishes exceptional cases from noisy samples
\item SHAP-FOLD learns a reasonable number of non-monotonic rules in the form of default theories % that would understandably capture the global behavior of any black-box model
\item SHAP-FOLD is fast and scalable compared to conventional ILP algorithms
\end{enumerate}
This paper makes the following novel contribution: We
present a new ILP algorithm capable of learning \textit{non-monotonic}
logic programs from local explanations of
black-box models provided by SHAP. Our experiments on UCI standard benchmark data sets suggest that SHAP-FOLD outperforms ALEPH \cite{aleph} in terms of classification evaluation metrics, running time, and providing more concise explanations measured in terms of number of clauses induced.
\fi
\iffalse
The rest of this paper is organized as follows: In Section \ref{sec:background} we present the problem definition as well as the background material to understand the SHAP-FOLD algorithm. In Section \ref{sec:SHAP-FOLD} we present the SHAP-FOLD algorithm. In Section \ref{sec:experiments} we present our experiments on UCI benchmark datasets. In Section \ref{sec:relatedworks} we discuss existing approaches and compare them with our SHAP-FOLD algorithm, and finally, in Section \ref{sec:conclusion} we present our conclusions with recommendations for future research.
\fi
\iffalse
\subsection{The FOIL Algorithm}
FOIL is a top-down ILP algorithm that follows a \textit{sequential covering} scheme to induce a hypotheses. The FOIL algorithm is summarized in Algorithm \ref{algo:foil}. This algorithm repeatedly searches for clauses that score best with respect to a subset of
positive and negative examples, a current hypothesis and a heuristic called \textit{information gain} (IG).
\fi
\iffalse
The inner loop searches for a clause with the highest information gain using a general-to-specific hill-climbing search. To specialize a given clause $c$, a refinement operator $\rho$ under $\theta$-subsumption \cite{plotkin70} is employed. The most general clause is the following:
$p(X_1,...,X_n) \gets true.$, where the predicate $p/n$ is the predicate being learned and each $X_i$ is a variable. The refinement operator specializes the current clause $h \gets b_1,...b_n .$ This is realized by adding a new literal $l$ to the clause yielding $h \gets b_1,...b_n,l$. The heuristic based search uses information gain.
\fi
\iffalse
\begin{algorithm}[!h]
\caption{Best Information Gain}
\label{algo:best_ig}
\begin{algorithmic}[1]
\Input $E^+,E^-,i,L_{used}$
\Output $best, l$ \Comment{}
\Function{Best\_info\_gain}{$E^+,E^-,i,L_{used}$}
\For{$e \in E^+$}
\State $pos[e_i] \gets pos[e_i] + 1$
\If{$e_i$ is numeric}
\State $xs \gets xs \cup e_i$
\State $xp \gets xp + 1$
\Else
\State $cs \gets cs \cup e_i$
\State $cp \gets cp + 1$
\EndIf
\EndFor
\For{$e \in E^-$}
\State $neg[e_i] \gets neg[e_i] + 1$
\If{$e_i$ is numeric}
\State $xs \gets xs \cup e_i$
\State $xn \gets xn + 1$
\Else
\State $cs \gets cs \cup e_i$
\State $cn \gets cn + 1$
\EndIf
\EndFor
\State $xs \gets xs.sort()$
\For{$j \gets 1 \quad to \quad xs.size$}
\State $pos[xs_{i}] \gets pos[xs_{i}] + pos[xs_{i-1}]$
\State $neg[xs_{i}] \gets neg[xs_{i}] + neg[xs_{i-1}]$
\EndFor
\State $best \gets -\infty$
\For{$x \in xs$}
\If{$literal(i,<=, x) \notin L_{used}$ \& $literal(i,>, x) \notin L_{used}$}
\State $g \gets Info\_gain(pos[x], xp-pos[x]+cp,xn-neg[x]+cn,neg[x])$
\If{$best < g$}
\State $best, l \gets g, literal(i,<=, x)$
\EndIf
\State $g \gets Info\_gain(xp-pos[x], pos[x]+cp,neg[x]+cn,xn-neg[x])$
\If{$best < g$}
\State $best, l \gets g, literal(i,>, x)$
\EndIf
\EndIf
\EndFor
\For{$c \in cs$}
\If{$literal(i,'==', x) \notin L_{used}$}
\State $g \gets Info\_gain(pos[c],cp-pos[c]+xp,cn-neg[c]+xn, neg[c])$
\If{$best < g$}
\State $best, l \gets g, literal(i,'==', x)$
\EndIf
\EndIf
\If{$literal(i,'!=', x) \notin L_{used}$}
\State $g \gets Info\_gain(cp-pos[c]+xp,pos[c],neg[c],cn-neg[c]+xn)$
\If{$best < g$}
\State $best, l \gets g, literal(i,'!=', x)$
\EndIf
\EndIf
\EndFor
\State \Return $best, l$
\EndFunction
\end{algorithmic}
\end{algorithm}
\fi
\iffalse
SHAP \cite{shap} (SHapley Additive exPlanations) is a unified approach with foundations in game theory to explain the output of any machine learning model.
in terms of its features' contributions. To compute each feature $i$'s contribution, SHAP requires retraining the model on all feature subsets $S \subseteq F$, where $F$ is the set of all features. For any feature $i$, a model $f_{S \cup \{i\}}$ is trained with the feature $i$ present, and another model $f_S$ is trained with feature $i$ eliminated. Then, the difference between predictions is computed as follows: $f_{S \cup \{i\}}(x_{S \cup \{i\}}) - f_S(x_S)$, where $x_S$ represents sample's feature values in $S$. Since the effect of withholding a feature depends on other features in the model, the above differences are computed for all possible subsets of $S \subseteq F \setminus \{i\}$ and their average taken. The weighted average of all possible differences (a.k.a Shapley value) is used as feature importance. The Equation \ref{eq:shapleynumber} shows how Shapley value associated with each feature value is computed:
\begin{equation}
\label{eq:shapleynumber}
\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F| - |S| -1)!}{|F|!} \bigg [f_{S \cup \{i\}}(x_{S \cup \{i\}}) - f_S(x_S) \bigg ]
\end{equation}
Given a dataset and a trained model, the SHAP framework computes a matrix of the shape $(\# samples, \#features)$ representing the Shapley value of each feature for each data sample. Each row sums to the difference between the model output for that sample and the expected value of the model output. This difference explains why the model is inclined to predict a specific class outcome.
\begin{exmp}
\label{ex:heart}
The UCI heart dataset contains features such as patient's blood pressure, chest pain, thallium test results, number of major vessels blocked, etc. The classification task is to predict whether the subject suffers from heart disease or not. Figure \ref{fig:heartshap} shows how SHAP would explain a model's prediction over a data sample. For this sample, SHAP explains why the model predicts heart disease by returning the top features along with their Shapley values (importance weight). According to SHAP, the model predicts ``heart disease" because of the values of ``thalium test" and ``maximum heart rate achieved" which push the prediction from the base (expected) value of 0.44 towards a positive prediction (heart disease). On the other hand, the feature ``chest pain" would have pushed the prediction towards negative (healthy), but it is not strong enough to turn the prediction.
\end{exmp}
\begin{figure}[h]
\centering
\includegraphics[scale=0.8]{shap_heart_example.png}
\caption{Shap Values for A UCI Heart Prediction }
\label{fig:heartshap}
\end{figure}
\fi
\iffalse
The categorical features should be \textit{binarized} before any model is trained. Binarization (a.k.a one-hot encoding) is the process of transforming each categorical feature with domain of cardinality $N$, into $N$ new binary predicates (features).In Example \ref{ex:heart}, chest pain level is a categorical feature with 4 different values in the set $\{1,2,3,4\}$. Type 4 chest pain indicates asymptomatic pain and is a serious indication of a heart condition. In this case, binarization results in 4 different predicates. The ``thalium test" is also a categorical feature with outcomes in the set \{3,6,7\}. Any outcome other than 3, indicates a defect (6 for fixed and reversible for 7). In case of Example \ref{ex:heart}, SHAP determines that the feature ``thal\_7" with outcome of 1 (True), pushes the prediction towards heart disease. To reflect this fact in our ILP algorithm, for any person $X$, the predicate \texttt{thal(X,7)} is introduced. Also, SHAP indicates that the binary feature ``chest\_pain\_4" with value 0 (False), pushes the prediction towards healthy.
\fi
\iffalse
\subsection{High-Utility Itemset Mining}
The problem of \textit{High-Utility Itemset Mining} (HUIM) is an extension of an older problem in data mining known as \textit{frequent pattern mining} \cite{freqpm}. Frequent pattern mining is meant to find frequent patterns in transaction databases. A \textit{transaction database} is a set of records (transactions) indicating the items purchased by customers at different times. A \textit{frequent itemset} is a group of items that appear in many transactions. For instance, \{noodles, spicy sauce\} being a frequent itemset, can be used to take marketing decisions such as co-promoting noodles with spicy sauce. Finding frequent itemsets is a well-studied problem with an efficient algorithm named Apriori \cite{apriori}. However, in some applications frequency is not always the objective. For example, the pattern \{milk,bread\} may be highly frequent, but it may yield a low profit. On the other hand, a pattern such as \{caviar, champagne\} may not be frequent but may yield a high profit. Hence, to find interesting patterns in data, other aspects such as profitability is considered.
Mining high utility itemsets can be viewed as a generalization of the frequent itemset mining where each item in each transaction has a utility (importance) associated with it and the goal is to find itemsets that generate high profit when for instance, they are sold together. The user has to provide a value for a threshold called \textit{minimum utility}. A high utility itemset mining algorithm outputs all the high-utility itemsets with at least \textit{minimum utility} profit.
\iffalse
The HUIM problem is formally defined as follows:
\begin{itemize}
\item $I = \{i_1, i_2,..., i_m\}$ is a set of items.
\item $D = \{T_1, T_2, ..., T_n\}$ be a transaction database where each transaction $T_i \in D$ is a subset of $I$.
\item $u(i_p,T_q)$ denotes the utility (profit) for item $i_p$ in transaction $T_q$. For example $u(b,T_0) = 10$ in Example from Table \ref{tbl:huiexample}.
\item $u(X,T_q)$, utility of an itemset $X$ is defined as $\sum_{i_p \in X}^{} u(i_p,T_q)$, where $X = \{i_1, i_2, ..., i_K\}$ is a k-itemset, $X \subseteq T_q$ and $1 \leq K \leq m$.
\item $u(X)$, utility of an itemset $X$ is defined as $\sum_{T_q \in D \land X \subseteq T_q} u(X,T_q)$.
\end{itemize}
An itemset $X$ is a \textit{high utility itemset} if $u(X) \geq \varepsilon$, where $X \subseteq I$ and $\varepsilon$ is the \textit{minimum utility} threshold, otherwise, it is a low utility itemset.
\fi
Table~\ref{tbl:huiexample} shows a transaction database consisting of 5 transactions. Left column shows the transaction Identifier. Middle column contains the items included in each transaction and right column contains each item's respective profit. If the \textit{minimum utility} is set to 25, the result of a high utility itemset mining algorithm is shown in the right table in Table \ref{tbl:huiexample}. In order to rank the high utility itemsets, Top-K High Utility Itemset (THUI) mining problem \cite{topk} is incorporated in SHAP-FOLD.
\iffalse
Several high utility itemset mining algorithms have been proposed (e.g., UMining, Two-Phase, IHUP, UP-Growth, etc). A complete survey of these algorithms can be found in \cite{huimsurvey}. The differences between these algorithms lies in the data structures, strategies that are employed for searching high utility itemsets (DFS vs. BFS), and how they prune the unpromising paths.
\fi
\begin{table}[]
\centering
\begin{tabular}{|c|l|l|}
\hline
\multicolumn{1}{|l|}{\textbf{Transactions}} & \textbf{Items} & \textbf{Profits} \\ \hline
$\mathbf{T_0}$ & a b c d e & 5 10 1 6 3 \\ \hline
$\mathbf{T_1}$ & b c d e & 8 3 6 3 \\ \hline
$\mathbf{T_2}$ & a c d & 5 1 2 \\ \hline
$\mathbf{T_3}$ & a c e & 10 6 6 \\ \hline
$\mathbf{T_4}$ & b c e & 4 2 3 \\ \hline
\end{tabular}
\quad
\begin{tabular}{|l|l|}
\hline
\multicolumn{2}{|c|}{\textbf{High Utility Itemsets}} \\ \hline
\{a, c\}: 28 & \{a, c, e\}: 31 \\
\{a, b, c, d, e\}: 25 & \{b, c\}: 28 \\
\{b, c, d\}: 34 & \{b, c, d, e\}: 40 \\
\{b, c, e\}: 37 & \{b, d\}: 30 \\
\{b, d, e\}: 36 & \{b, e\}: 31 \\
\{c, e\}: 27 & \\ \hline
\end{tabular}
\caption{Left: An HUIM Problem Instance. Right: Solution for minutil = 25 }
\label{tbl:huiexample}
\end{table}
\iffalse
which works as follows: It initially sets an internal $\varepsilon$ threshold to 0, and starts to explore the search space. Then, as soon as k high utility itemsets are found, the internal $\varepsilon$ is raised to the utility of the pattern having the lowest utility among the current top-k patterns. Then, the search continues and for each high utility itemset found, the set
of the current top-k pattern is updated as well as the internal $\varepsilon$.
When the algorithm terminates, it returns the set of the top-k high utility itemsets.
\fi
\fi
\iffalse
\section{SHAP-FOLD Algorithm}
\label{sec:SHAP-FOLD}
SHAP-FOLD learns a concept in terms of a default theory \cite{fold}. In Logic Programming, default theories are represented using \textit{negation-as-failure} (NAF) semantics \cite{Baral}.
\begin{exmp}
\label{ex:flies}
The following default theory ``Normally, birds fly except penguins which do not", is represented as:
\begin{verbatim}
flies(X) :- bird(X), not ab_bird(X).
ab_bird(X) :- penguin(X).
\end{verbatim}
\end{exmp}
The SHAP-FOLD algorithm adapts the FOIL style sequential covering scheme. Therefore, it iteratively learns single clauses, until all positive examples are covered. To learn one clause, SHAP-FOLD first finds common patterns among positive examples. If the resulted clause (default) covers a significant number of negative examples, SHAP-FOLD swaps the current positive and negative examples and recursively calls the algorithm to learn common patterns in negative examples (exceptions). As shown in Example \ref{ex:flies}, the exceptions are ruled out using negation-as-failure. Learning exceptions allow our SHAP-FOLD algorithm to distinguish between noisy samples and exceptional cases.
\begin{algorithm}[h]
\caption{Summary of SHAP-FOLD Algorithm}
\label{algo:SHAP-FOLD}
\begin{algorithmic}[1]
\Input ~~~$G$: Target Predicate to Learn
\Statex ~~~~~~~~~$B$: Background Knowledge
\Statex ~~~~~~~~~$D ~~= \{~ (\vec{x}_1,y_1),...,(\vec{x}_m,y_m) \}$ : ~$y_i \in \{-1,+1\}$
\Statex ~~~~~~~~~$E^+ = \{ ~\vec{x}_i ~|~ \vec{x}_i \in D \land y_i = 1\}$ :~ Positive Examples
\Statex ~~~~~~~~~$E^- = \{ ~\vec{x}_i ~|~ \vec{x}_i \in D \land y_i = -1\}$:~ Negative Examples
\Statex ~~~~~~~~~$T ~~~= \{~(\vec{z}_i, \vec{\phi}_i) ~|~ \vec{z}_i \subseteq \vec{x}_i \land \vec{x}_i \in D \land \vec{\phi}_i$ is $\vec{z}_i$'s Shapley values $\}$
\Output $D ~~~= \{~C_1,...,C_n\}$ \Comment{default clauses}
\Statex ~~~~~~$AB ~~~= \{~ab_1,...,ab_m\}$ \Comment{exceptions/abnormal
clauses}
\Function{FOIL}{$E^+,E^-$}
\While{($|E^+| > 0$)}
\State $C_{def+exc} \gets$ \Call{Learn\_One\_Rule}{{$E^+$},{{$E^-$}}}
\State $E^+ \gets E^+ \setminus covers(C_{def+exc},E^+,B)$
\State $D \gets D \cup \{ C_{def+exc} \}$
\EndWhile
\State \textbf{return} $D, AB$ \Comment{returns sets of defaults and exceptions}
\EndFunction
\Function{Learn\_One\_Rule}{${E^+},{E^-}$}
\State - let Item-Set be $\{(f_1,...f_n),(\phi_1,...,\phi_n)\} \gets$ \Call{Top-K}{K=1,{$E^+$},{T}} %\Comment{Call to HUIM algorithm}
\State $C_{def} \gets (G$ :- $f_1,...,f_n)$
\State $FP \gets covers(C_{def},E^-)$ \Comment{FP denotes False Positives}
\If{$FP > 0$}
\State $C_{def+exc} \gets \Call{LEARN\_EXCEPTIONS}{{C_{def}},{E^-},{{E^+}}}$
\EndIf
\State \textbf{return} $C_{def+exc}$
\EndFunction
\Function{LEARN\_EXCEPTIONS}{${C_{def}},{E^+},{E^-}$}
\State $ \{C_1,...,C_k\} \gets \Call{FOIL}{E^+,E^-} $ \Comment{Recursive Call After Swapping}
\State $ab\_index \gets GENERATE\_UNIQUE\_AB\_INDEX()$
\For {$ i \gets 1$ \textbf{to} $k$}
\State $AB \gets AB \cup \{ ab_{ab\_index} \ $:-$ \ bodyof(C_i) \}$
\EndFor
\State \textbf{return} $C_{def+exc} \gets (headof(C_{def}) \ $:-$ \ bodyof(C_{def}), \ \textbf{not}(ab_{ab\_index}))$
\EndFunction
\end{algorithmic}
\end{algorithm}
To search for ``best" clause, SHAP-FOLD tightly integrates the High Utility Itemset Mining (HUIM) and the SHAP technique. In this novel approach, the SHAP system is employed to find relevant features as well as their importance. To find the ``best" clause SHAP-FOLD creates instances of HUIM problem. Each instance, contains a subset of examples represented as a set of ``transactions" as shown in Table \ref{tbl:huiexample}. Each ``transaction" contains a subset of feature values along with their corresponding utility (i.e., feature importance). The feature importance $\phi_i \in [0,1]$ for all $i$ distinct feature values. Therefore, a \textit{high-utility itemset} in any set of ``transactions" represents strongest features that would contribute to the classification of a significant number of examples, because, otherwise, that itemset would not have been selected as a high-utility itemset. To find the itemset with highest utility, the HUIM algorithm Top-K \cite{topk} is invoked with $K$ set to 1.
SHAP-FOLD takes a target predicate name (G), a tabular dataset (D) with $m$ rows and two different labels $+1$ and $-1$ for positive examples and negative examples respectively. $E^+$ and $E^-$ represent these examples in the form of target atoms. It also takes a ``transaction" database. Each row of T contains a subset of an example's feature-values ($\vec{z}_i$) along with their Shapley values ($\vec{\phi}_i$). This ``transaction" database is passed along to create HUIM instance and find the itemset with highest utility every time Top-K algorithm is invoked. The summary of SHAP-FOLD's pseudo-code is shown in Algorithm \ref{algo:SHAP-FOLD}.
In the function FOIL (lines 1-8), \textit{sequential covering } loop to cover positive examples is realized. On every iteration, a default clause (and possibly multiple exceptions) - denoted by $C_{def+exc}$ - is learned and added to the hypothesis. Then, the covered examples are removed from the remaining examples. In the function LEARN\_ONE\_RULE (lines 9-17), Top-K algorithm with $k=1$ is invoked and a high-utility itemset (i.e., a subset of features-values and their corresponding Shapley values) is retrieved. These subset of features create the default part of a new clause. Next, if the default clause covers false positives, the current positive and negative examples are swapped to learn exceptions. In the function LEARN\_EXCEPTIONS (lines 18 - 25), the algorithm recursively calls itself to learn clauses that would cover exceptional patterns. When the recursive call returns, for all learned clauses, their head is replaced by an abnormality predicate. To manufacture the complete default theory, the abnormality predicate preceded by negation-as-failure (not) is added to the default part. Example \ref{ex:car} shows how SHAP-FOLD learns a concise logic program from an XGBoost trained model.
\begin{exmp}
\label{ex:car}
The ``UCI Cars" dataset contains 1728 different cars and their acceptability based on features such as buying price, maintenance cost, trunk size, capacity, number of doors, and safety. SHAP-FOLD generates the following program from a trained XGBoost model:
\end{exmp}
\begin{verbatim}
DEF(1): acceptable(A):- safety(A,high), not ab0(A).
EXCEPTIONS(1): ab0(A):- persons(A,2).
ab0(A):- maintenance(A,very_high).
DEF(2): acceptable(A):- persons(A,4), safety(A,medium), not ab1(A).
EXCEPTIONS(2): ab1(A):- price(A,very_high), trunk(A,small).
ab1(A):- price(A,high), maintenance(A,very_high).
DEF(3): acceptable(A):- trunk(A,big), safety(A,medium),
persons(A,>5).
\end{verbatim}
On first iteration, the clause DEF(1) is generated. Since it covers a significant number of negative examples, $E^+$ and $E^-$ are swapped and algorithm recursively calls itself. Inside LEARN\_EXCEPTIONS, the recursive call returns with EXCEPTIONS(1) clauses. The head predicate \texttt{ab0} replaces their head and finally in line 24, the negation of abnormality is appended to the default to create the following default clause: ``A car is considered acceptable if its safety is high, unless it only fits two persons or its maintenance cost is high".
\iffalse
Similarly, the DEF(2) clause states that a car is acceptable if it can fit 4 person and its safety is medium, unless its price is too high and its trunk is small. Another exceptional case is established by high price and very high maintenance. The third clause default part does not cover any false positives, hence no exception clause is learned.
\fi
\iffalse
There are some technicalities that should be pointed out: (1) The numeric features should be discretized (i.e., by splitting them into fixed number of intervals). This restriction is imposed by SHAP technique. (2) Inspired by FOIL implementation, the ``if statement" in line 13 of the algorithm is realized in terms of an empirical accuracy (e.g., \% 85) to avoid over-fitting. This would allow some noise error after learning the exceptions.
\fi
\fi
\iffalse
The SHAP-FOLD requires a statistical model as input to the SHAP technique. While computing the Shapley values is slow, there is a fast and exact implementation called TreeExplainer \cite{treeshap} for ensemble tree models. XGBoost \cite{xgboost} is a powerful ensemble tree model that perfectly works with TreeExplainer. Thus, we trained an XGBoost model for each of the reported experiments in this paper. Table \ref{tbl:accuracies} presents the comparison between ALEPH and SHAP-FOLD on classification evaluation of each UCI dataset. The best performer is highlighted with boldface font. In terms of the running time, SHAP-FOLD scales up much better. In case of ``King-Rook vs. King-Pawn", while ALEPH discovers 283 clauses in 836 seconds, SHAP-FOLD does much better. It finishes in 8 seconds discovering only 3 clauses that cover the knowledge underlying the model. Similarly, in case of ``UCI kidney", SHAP-FOLD finds significantly fewer clauses.Thus, not only SHAP-FOLD's performance is much better, it discovers more succinct programs. Also, scalability is a major problem in ILP, that our SHAP-FOLD algorithm solves: its execution performance is orders of magnitude better.
\fi
\iffalse
SHAP-FOLD almost always achieves a higher Recall score.This suggests that the proper use of \textit{negation-as-failure} leads to better coverage. The absence of negation from ALEPH hypothesis space forces the algorithm to create too specific clauses which leaves many positive examples uncovered. In contrast, our SHAP-FOLD algorithm emphasizes on better coverage via finding high-utility patterns of important features first. If the result turns out to cover too many negative examples to tolerate, by learning exceptions and ruling them out (via the same algorithm applied recursively), SHAP-FOLD maintains the same coverage as it rules out exceptional negative examples.
\fi
\iffalse
SHAP-FOLD is a Java application that interfaces SWI-Prolog \cite{swiweb} using JPL library. The HUIM instances are solved by calling TKU from the SPMF Data mining library \cite{spmf}.
\fi
\iffalse
{\scriptsize
\begin{verbatim}
ANSWER: 1 (in 3.673 ms)
JUSTIFICATION_TREE:
'survived' holds (for 924, and 1), because
person 924 is female, and
there is no evidence that 'class' holds (for 924, and 3), and
there is no evidence that abnormal case 2 holds for 924, because
there is no evidence that person 924 paid Var1 not equal 20.575 for the ticket, and
person 924 paid 20.575 for the ticket, and
person 924 paid 20.575 for the ticket, justified above.
The global constraints hold.
\end{verbatim}
}
\fi
\begin{thebibliography}{10}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\providecommand{\doi}[1]{https://doi.org/#1}
\bibitem{arias-justification}
Arias, J., Carro, M., Chen, Z., Gupta, G.: Justifications for goal-directed
constraint answer set programming. In: Proceedings 36th International
Conference on Logic Programming (Technical Communications). {EPTCS},
vol.~325, pp. 59--72 (2020)
\bibitem{scasp}
Arias, J., Carro, M., Salazar, E., Marple, K., Gupta, G.: Constraint answer set
programming without grounding. Theory and Practice of Logic Programming
\textbf{18}(3-4), 337--354 (2018)
\bibitem{Baral}
Baral, C.: Knowledge representation, reasoning and declarative problem solving.
Cambridge University Press, Cambridge, New York, Melbourne (2003)
\bibitem{xgboost}
Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In:
Proceedings of the 22Nd ACM SIGKDD. pp. 785--794. KDD '16 (2016)
\bibitem{trepan}
Craven, M.W., Shavlik, J.W.: Extracting tree-structured representations of
trained networks. In: Proceedings of the 8th International Conference on
Neural Information Processing Systems. pp. 24--30. NIPS'95, MIT Press,
Cambridge, MA, USA (1995)
\bibitem{rulefit}
Friedman, J.H., Popescu, B.E., et~al.: Predictive learning via rule ensembles.
The Annals of Applied Statistics \textbf{2}(3), 916--954 (2008)
\bibitem{gelfondkahl}
Gelfond, M., Kahl, Y.: Knowledge representation, reasoning, and the design of
intelligent agents: The answer-set programming approach. Cambridge University
Press (2014)
\bibitem{xai}
Gunning, D.: Explainable artificial intelligence (xai) (2015),
\url{\url{https://www.darpa.mil/program/explainable-artificial-intelligence}}
\bibitem{nfoil}
Landwehr, N., Kersting, K., Raedt, L.D.: n{FOIL}: Integrating na{\"{\i}}ve
bayes and {FOIL}. In: Proceedings, The Twentieth National Conference on
Artificial Intelligence and the Seventeenth Innovative Applications of
Artificial Intelligence Conference, July 9-13, 2005, Pittsburgh,
Pennsylvania, {USA}. pp. 795--800 (2005)
\bibitem{kfoil}
Landwehr, N., Passerini, A., Raedt, L.D., Frasconi, P.: k{FOIL}: Learning
simple relational kernels. In: Proceedings, The Twenty-First National
Conference on Artificial Intelligence and the Eighteenth Innovative
Applications of Artificial Intelligence Conference, July 16-20, 2006, MA,
{USA}. pp. 389--394 (2006)
\bibitem{uci}
Lichman, M.: {UCI}, {M}achine {L}earning {R}epository, \url{
http://archive.ics.uci.edu/ml} (2013)
\bibitem{ilp}
Muggleton, S.: Inductive logic programming. New Gen. Comput. \textbf{8}(4)
(Feb 1991)
\bibitem{svmilp}
Muggleton, S., Lodhi, H., Amini, A., Sternberg, M.J.E.: Support vector
inductive logic programming. In: Hoffmann, A., Motoda, H., Scheffer, T.
(eds.) Discovery Science. Springer Berlin Heidelberg, Berlin, Heidelberg
(2005)
\bibitem{ilp20}
Muggleton, S., Raedt, L., Poole, D., Bratko, I., Flach, P., Inoue, K.,
Srinivasan, A.: Ilp turns 20. Mach. Learn. \textbf{86}(1), 3--23 (Jan 2012)
\bibitem{svmplus}
Núñez, H., Angulo, C., Català, A.: Rule extraction from support vector
machines. In: In Proceedings of European Symposium on Artificial Neural
Networks. pp. 107--112 (2002)
\bibitem{plotkin70}
Plotkin, G.D.: A further note on inductive generalization, in machine
intelligence, volume 6, pages 101-124 (1971)
\bibitem{foil}
Quinlan, J.R.: Learning logical definitions from relations. Machine Learning
\textbf{5}, 239--266 (1990)
\bibitem{sakama05}
Sakama, C.: Induction from answer sets in nonmonotonic logic programs. {ACM}
Trans. Comput. Log. \textbf{6}(2), 203--231 (2005)
\bibitem{shakerin-phd}
Shakerin, F.: Logic Programming-based Approaches in Explainable AI and Natural
Language Processing. Ph.D. thesis (2020), {D}epartment of Computer Science,
The University of Texas at Dallas
\bibitem{fold}
Shakerin, F., Salazar, E., Gupta, G.: A new algorithm to automate inductive
learning of default theories. {TPLP} \textbf{17}(5-6), 1010--1026 (2017)
\bibitem{aleph}
Srinivasan, A.: The Aleph Manual, \\
\url{https://www.cs.ox.ac.uk/activities/programinduction/Aleph/aleph.html}
(2001)
\bibitem{Akihiro2021}
Takemura, A., Inoue, K.: Generating explainable rule sets from tree-ensemble
learning methods by answer set programming. Electronic Proceedings in
Theoretical Computer Science \textbf{345}, 127–140 (Sep 2021).
\doi{10.4204/eptcs.345.26}, \url{http://dx.doi.org/10.4204/EPTCS.345.26}
\bibitem{prefixsum}
{Wikipedia contributors}: Prefix sum {Wikipedia}{,} the free encyclopedia
(2021), \url{\url{https://en.wikipedia.org/wiki/Prefix\_sum}}, online;
accessed 5 October, 2021
\bibitem{quickfoil}
Zeng, Q., Patel, J.M., Page, D.: Quickfoil: Scalable inductive logic
programming. Proc. VLDB Endow. \textbf{8}(3), 197--208 (Nov 2014)
\end{thebibliography}