CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models
Extracted source text · arXiv 1905.07857
%%%% ijcai19.tex
\typeout{IJCAI-19 Instructions for Authors}
% These are the instructions for authors for IJCAI-19.
\documentclass{article}
\pdfpagewidth=8.5in
\pdfpageheight=11in
% The file ijcai19.sty is NOT the same than previous years'
\usepackage{ijcai19}
% Use the postscript times font!
\usepackage{times}
\usepackage[colorlinks]{hyperref}
\usepackage{soul}
\usepackage{url}
%\usepackage[hidelinks]{hyperref}
\usepackage[utf8]{inputenc}
\usepackage[small]{caption}
\usepackage{graphicx}
\usepackage{amsmath,amssymb}
\usepackage{mathtools}
\usepackage{booktabs}
\usepackage{algorithm}
\usepackage{algorithmic}
\urlstyle{same}
\newcommand{\methodName}{{CERTIFAI}}
\usepackage{tikz}
\newcount\Comments % 0 suppresses notes to selves in text
\Comments=1% TODO: set to 0 for final version
\usepackage{color}
\definecolor{purple}{rgb}{1,0,1}
\definecolor{orange}{rgb}{.85,.25,0}
\newcommand{\kibitz}[2]{\ifnum\Comments=1\textcolor{#1}{#2}\fi}
\newcommand{\jette}[1]{\kibitz{orange} {[Jette: #1]}}
\newcommand{\shubham}[1]{\kibitz{blue} {[Shubham: #1]}}
% the following package is optional:
%\usepackage{latexsym}
% Following comment is from ijcai97-submit.tex:
% The preparation of these files was supported by Schlumberger Palo Alto
% Research, AT\&T Bell Laboratories, and Morgan Kaufmann Publishers.
% Shirley Jowell, of Morgan Kaufmann Publishers, and Peter F.
% Patel-Schneider, of AT\&T Bell Laboratories collaborated on their
% preparation.
% These instructions can be modified and used in other conferences as long
% as credit to the authors and supporting agencies is retained, this notice
% is not changed, and further modification or reuse is not restricted.
% Neither Shirley Jowell nor Peter F. Patel-Schneider can be listed as
% contacts for providing assistance without their prior permission.
% To use for other conferences, change references to files and the
% conference appropriate and use other authors, contacts, publishers, and
% organizations.
% Also change the deadline and address for returning papers and the length and
% page charge instructions.
% Put where the files are available in the appropriate places.
\title{\methodName: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models}
% Single author syntax
% Multiple author syntax (remove the single-author syntax above and the \iffalse ... \fi here)
% Check the ijcai19-multiauthor.tex file for detailed instructions
\author{
Shubham Sharma$^1$
\and
Jette Henderson$^2$\and
Joydeep Ghosh$^2$
\affiliations
$^1$Department of Electrical and Computer Engineering, University of Texas at Austin\\
$^2$CognitiveScale\\
\emails
shubham\_sharma@utexas.edu,
jhenderson@cognitivescale.com,
jghosh@utexas.edu
}
%\author{
%Responsible Artificial Intelligence
%\and
%Interpretable Machine Learning\and
%Robustness
%\affiliations
%}
\begin{document}
\maketitle
\begin{abstract}
As artificial intelligence plays an increasingly important role in our society, there are ethical and moral obligations for both businesses and researchers to ensure that their machine learning models are designed, deployed, and maintained responsibly.
These models need to be rigorously audited for fairness, robustness, transparency, and interpretability.
A variety of methods have been developed that focus on these issues in isolation, however, managing these methods in conjunction with model development can be cumbersome and time-consuming.
In this paper, we introduce a unified and model-agnostic approach to address these issues: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models (CERTIFAI). Unlike previous methods in this domain, \methodName~is a general tool that can be applied to any black-box model and any type of input data.
Given a model and an input instance, \methodName~uses a custom genetic algorithm to generate counterfactuals\footnote{ Counterfactuals have a well-established meaning in the causality literature. However, we are using “counterfactual” in the counterfactual explanation sense, one that has been recently initiated in the explainability literature \cite{wachter2017counterfactual} where the model implies a machine learning model and not a causal model and hence no causal assumptions are made here}: instances close to the input that change the prediction of the model.
We demonstrate how these counterfactuals can be used to examine issues of robustness, interpretability, transparency, and fairness. Additionally, we introduce CERScore, the first black-box model robustness score that performs comparably to methods that have access to model internals.
\end{abstract}
\section{Introduction}
%As machine learning grows rapidly, it becomes important to think of the ethical, moral and social consequences that the decisions made by such model have on society.
As the adoption of machine learning models to everyday tasks grows rapidly, so does the need to consider the ethical, moral, and social consequences of the decisions made by such models.
Several important questions arise in a variety of applications, such as the following: 1) how did the model predict what it predicted?, 2) if a person got an unfavorable outcome from the model, what can they do to change that?, 3) has the model been unfair to a particular group?, and 4) how easily can the model be fooled?
Researchers are actively building separate approaches to answer each of these questions. %However, these existing approaches require making assumptions on the type of model~\cite{ustun2019actionable} and the type of data~\cite{wachter2017counterfactual} for which they can be used or they require access to the model weights to provide explainability~\cite{bach2015pixel} and robustness~\cite{weng2018evaluating}.
%While there are a diverse set of these approaches dealing with different aspects (fairness, explainability, robustness) to build responsible and trustworthy models.\jette{cite more methods/approaches}, there is no single tool or algorithm that can answer all the questions above. This paper introduces a unified approach \methodName: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models to address this gap.
\begin{figure}[tp]
\centering
\includegraphics[width = 8.5cm]{counterfactual2.PNG}
\caption{The \methodName~counterfactual generation process. The decision boundary for a binary classifier is shown, with the input instance in black. We sample a set of points (left) in the feature space with a constraint that they must lie on the other side of the decision boundary (green points). The algorithm then evolves these samples (middle) to generate individuals that lie closer to the input point but on the other side of the decision boundary. Finally, a smaller set, the size of which is user-defined, of counterfactuals is generated (right).}
\label{fig:CE_example}
\end{figure}
% To answer these questions, researchers are building separate tools and algorithms to explain a machine learning model, to decide if it is fair, and to gauge how robust it is to adversarial attacks. However, these existing approaches require making assumptions on the type of model~\cite{ustun2019actionable} and the type of data~\cite{wachter2017counterfactual} for which they can be used or they require access to the model weights to provide explainability~\cite{bach2015pixel} and robustness~\cite{weng2018evaluating}. While there are a diverse set of these approaches dealing with different aspects (fairness, explainability, robustness) to build responsible and trustworthy models.\jette{cite more methods/approaches}, there is no single tool or algorithm that can answer all the questions above. This paper introduces a unified approach \methodName: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models to address this gap.
%Given a black-box machine learning model (i.e. we only have access to the prediction for any input we give to the model) and an input data point, a counterfactual is defined as a generated data point that is as close to the input data point as possible for which the model gives a different outcome.
\begin{table*}[ht]
\def\checkmark{\tikz\fill[scale=0.4](0,.35) -- (.25,0) -- (1,.7) -- (.25,.15) -- cycle;}
\centering
\begin{tabular}{lrrrrrrrr}
\toprule
Method & Black-box & Model-Agnostic & Mixed-data & Explainability & Fairness & Robustness\\
\midrule
CERTIFAI & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark \\
\cite{ustun2019actionable} & \checkmark & & \checkmark & \checkmark & \checkmark & \\
\cite{wachter2017counterfactual} & \checkmark & \checkmark & & \checkmark & \checkmark & \\
\cite{russell2019efficient} & \checkmark & & \checkmark & \checkmark &\checkmark & \\
\cite{ribeiro2016should} & \checkmark & \checkmark & \checkmark & \checkmark & & \\
\cite{guidotti2018local} & \checkmark & \checkmark & \checkmark & \checkmark & & \\
\cite{carlini2017towards} & & \checkmark & & & & \checkmark \\
\cite{weng2018evaluating} & & \checkmark & & & & \checkmark \\
\bottomrule
\end{tabular}
\caption{Comparison of related work with our approach. We consider the approach most similar to ours. Mixed-data means the method can work with both discrete and continuous data, without any discretization or assumptions.}
\label{tab:relwork}
\end{table*}
One promising vein of research in explainability, first introduced by \cite{wachter2017counterfactual}, is generating counterfactuals.
Given an input data point and a black-box machine learning model (i.e. we only have access to the model's prediction for any input), a \emph{counterfactual} is defined as a generated data point that is as close to the input data point as possible but for which the model gives a different outcome.
For example, if a user was denied a loan by a machine learning model, an example counterfactual explanation could be: ``Had your income been \$5000 greater per year and your credit score been 30 points higher, your loan would be approved."
\cite{wachter2017counterfactual} argue that counterfactuals are a way of explaining model results to users such that they can identify actionable ways of changing their behaviors to obtain favorable predictions.
In addition to providing counterfactuals for explainability, we show how counterfactual explanations can be used to audit fairness and robustness of a model.
%For example, if a user was denied a loan by a machine learning model, an example counterfactual explanation could be: "Had your income been \$60000 and your credit score been 710, your loan would've been approved".
As promising as the original method~\cite{wachter2017counterfactual} and subsequent methods of generating counterfactuals~\cite{ustun2019actionable,russell2019efficient} are, they are also limited in that some only work for linear models, while others cannot deal with different data types.
%Furthermore, no existing approaches use counterfactuals to explore issues in robustness.
To resolve these limitations, we introduce \methodName, a novel, flexible, model-agnostic technique for generating counterfactuals via a custom genetic algorithm.
The meta-heuristic evolutionary algorithm starts by generating a random set of points such that they do not have the same prediction as the input point.
A subsequent evolutionary process results in a set of points close to the input that maintain the prediction constraint.
Figure~\ref{fig:CE_example} shows an example of three counterfactuals (green points) generated for a given input (black point).
%Add to theory: Consider a binary classifier with a given data-point for which we need to generate a counterfactual, as shown in figure 1, where the black point represents the input. A set of points is sampled on the other side of the decision boundary, represented by the green points in the figure. The genetic algorithm is then used to find the k-closest points to the input data point, constrained on them remaining on the other side of the decision boundary. The sampled points evolve to generate the k set counterfactuals, as shown in the figure (k=3).
A major advantage of using the genetic algorithm to generate counterfactuals is that it can generate counterfactuals for linear and non-linear models (e.g. deep networks) and for any input form (from mixed tabular data to image data) without any approximations to or assumptions for the model.
Moreover, end-users can 1) define a range for any feature, and 2) restrict the features that can change.
\methodName~simply constrains the values of the sampled points based on those choices, allowing the generated counterfactuals to reflect a user's understanding of how much it is \emph{possible} for them to change their features.
% In this paper, we generate such counterfactuals and demonstrate how their uses.
% First, counterfactuals can be used to explain to person, the minimum change that they needed to get a different outcome from a model.
% Second, counterfactual generation can be used to assess the fairness of a model.
% For a sensitive attribute (such as race or gender), if the final distance of the counterfactual generated is much farther for a particular group (defined by the sensitive attribute value) than it is for other groups, given that all the instances received unfavorable outcomes, then this suggests the model has a bias towards certain groups.
% Third, counterfactual images can be used as adversarial examples. A number of images and their counterfactuals can be used to get an overall robustness score of the model, since a more robust model would have counterfactuals that look less like their corresponding inputs (and hence are less prone to adversarial attacks).
\methodName~can be used to audit any black-box classifier, providing three tools that are based on a single underlying algorithm. The major contributions of this paper are summarized as follows:
\begin{itemize}
\item Counterfactuals are generated using a custom genetic algorithm, which is model-agnostic, flexible, and can be used to provide explanations.
\item Counterfactuals are shown to be effective adversarial examples. They are also used to generate the Counterfactual Explanation-based robustness scores (CERScore), which to the best of our knowledge is the first ever black-box model robustness score.
\item Counterfactuals can be used to evaluate fairness with respect to a user's input as well as the fairness of the model towards groups of individuals.
\end{itemize}
\section{Related Work}
Table~\ref{tab:relwork} summarizes the key features of \methodName~and the work most related to~\methodName. Other general methods on explainability, fairness and robustness have been described by \cite{guidotti2018survey},\cite{binns2017fairness}, and \cite{akhtar2018threat} respectively. As we can see, there is no prior art that handles a diverse set of desirable properties needed to develop a responsible AI system. \cite{wachter2017counterfactual} introduced counterfactual explanations, however, their optimization formulation cannot handle categorical data (i.e. the optimization does not solve for such data; the values are found in a brute-force fashion).
Methods in \cite{ustun2019actionable} and \cite{russell2019efficient} only work for linear models. \cite{guidotti2018local} uses a genetic algorithm to generate neighbors around the input and then use decision trees to locally approximate the model.
However, local approximations might be at the cost of model accuracy, and they define counterfactuals based on the minimum number of splits in the trained decision tree, which might not always be the smallest change to the input.
The closest work to generating adversarial examples is by \cite{carlini2017towards}.
They work on a white-box and simpler convolution models, and the distance metrics might not be apt to measure image similarity. \cite{wang2002image}.
\cite{weng2018evaluating} define a robustness score CLEVER. However, they have access to the model gradients.
%This paper not only presents the first unified framework to audit algorithms, it is also the first to define a robustness score for black-box models.
% Counterfactual explanations, first introduced by ~\cite{wachter2017counterfactual}, show an actionable way of explaining the results of a model to a user, and we will also show how they can be used to examine issues of fairness and robustness.
% Given an input data point and black-box machine learning model (i.e. we only have access to the prediction for any input we give to the model), a counterfactual is defined as a generated data point that is as close to the input data point as possible but for which the model gives a different outcome.
% For example, if a user was denied a loan by a machine learning model, an example counterfactual explanation could be: "Had your income been \$5000 greater per year and your credit score been 30 points higher, your loan would be approved."
% %For example, if a user was denied a loan by a machine learning model, an example counterfactual explanation could be: "Had your income been \$60000 and your credit score been 710, your loan would've been approved".
% The original method~\cite{wachter2017counterfactual} and subsequent methods of generating counterfactuals~\cite{ustun2019actionable,russell2019efficient} are limited in that they only work for linear models or they do not non-numerical data types.
% \methodName~is a flexible, model-agnostic technique for generating counterfactuals that uses a genetic algorithm, which is a meta-heurestic evolutionary algorithm.
% The genetic algorithm samples points in the feature space such that the prediction for these points are different from an input point prediction, and the set of selected points then evolve based on them having the least distance to the input data point, while still being constrained on not belonging to the same class as the observed input.
% Figure~\ref{fig:CE_example} shows an example of three counterfactuals (green points) generated for a given input (black point).
% %Add to theory: Consider a binary classifier with a given data-point for which we need to generate a counterfactual, as shown in figure 1, where the black point represents the input. A set of points is sampled on the other side of the decision boundary, represented by the green points in the figure. The genetic algorithm is then used to find the k-closest points to the input data point, constrained on them remaining on the other side of the decision boundary. The sampled points evolve to generate the k set counterfactuals, as shown in the figure (k=3).
% A major advantage of using such an evolutionary approach is that we do not need to restrict ourselves to a particular class of models or input types. The algorithm can generate counterfactuals for linear and non-linear models (e.g. deep networks) and for any input form (from tabular data to image data).
% Moreover, end-users can define a range for any feature within which the feature is allowed to change. For example, a person denied a loan may specify that is only reasonable for their income to increase by \$1000 per year.
% A user can also restrict the features that can change.
% For example, a user may specify that job type, age, race, or gender cannot change in the evolution process.
% \methodName~simply constrains the values of the sampled points based on those choices.
% In this paper, we generate such counterfactuals and demonstrate how their uses.
% First, counterfactuals can be used to explain to person, the minimum change that they needed to get a different outcome from a model.
% Second, counterfactual generation can be used to assess the fairness of a model.
% For a sensitive attribute (such as race or gender), if the final distance of the counterfactual generated is much farther for a particular group (defined by the sensitive attribute value) than it is for other groups, given that all the instances received unfavorable outcomes, then this suggests the model has a bias towards certain groups.
% Third, counterfactual images can be used as adversarial examples. A number of images and their counterfactuals can be used to get an overall robustness score of the model, since a more robust model would have counterfactuals that look less like their corresponding inputs (and hence are less prone to adversarial attacks).
% \methodName~ can be used to audit any black-box classifier, providing three tools that are a result of a single algorithm. The major contributions of this paper are summarized as follows:
% \begin{itemize}
% \item Counterfactuals are generated using a genetic algorithm based approach, which is model-agnostic and flexible
% \item Counterfactuals are shown to be perfect adversarial examples and they are used to generate CERScore: Counterfactual based robustness score, which is the first ever black-box model robustness score.
% \item We demonstrate how a counterfactual or a set of counterfactuals can be used to explain a model's prediction
% \item We show the use of counterfactuals to evaluate fairness with respect to a user's input and the fairness of the model towards a groups of individuals
% \end{itemize}
%\section{Related Work}
%Research on Responsible Artificial Intelligence (RAI), with a focus on explainability, fairness and robustness of machine learning models and the ethical, moral and legal consequences while dealing with AI has been growing rapidly.
% Research focusing on explainability, fairness, and robustness of machine learning models and the ethical, moral, and legal consequences while dealing with AI has been growing rapidly.
% To the best of our knowledge, there is no unified approach that addresses interpretability, robustness and fairness in the literature. Different algorithms have been developed to address different concerns with fairness, interpretability and robustness of machine learning models.
%In this section, we discuss and compare the literature for analying the robustness, fairness, and interpretability of machine learning models with a particular focus on counterfactuals.% and past work in their use for each of the three applications.
%\subsection{Explainability}
%\emph{Explainability.}
%As machine learning models have grown more complicated, there has been a push in research to develop methods to explain or interpret these methods. Explainability can be with respect to the entire model or with respect to a particular input to the model. In this paper, we focus on explaining the prediction of a model for a particular instance by providing a set of counterfactual explanations and discuss similar approaches in this section. Other popular approaches to explain a model include: building a surrogate, more explainable model \cite{tan2017detecting,lou2013accurate,lakkaraju2017interpretable}, explaining training instances by using model weights, such as in \cite{bach2015pixel} and \cite{dhurandhar2018explanations} and locally approximating a model by training another model near an input as in \cite{ribeiro2016should,ribeiro2018anchors,guidotti2018local}.
%\cite{wachter2017counterfactual} demonstrated the use of counterfactuals for explaining machine learning models by posing their generation as an optimization problem.
%They also show how counterfactuals showcase the bias towards certain groups.
%However, there are several drawbacks to their method. In practice, their algorithm does not work well with categorical variables in the data.
%If the input in question has categorical variables, they run the optimization for every possible combination of categorical variable values. Additionally, they do not delve deeper into the possible use of counterfactuals for fairness.
%The What-if tool by Google builds on this approach, but instead of solving the optimization problem, they find the closest point within the training dataset that changes the prediction. That point might not necessarily be the smallest change a user can make, and this requires access to all the training data.
%Other approaches, such as the work by~\cite{ustun2019actionable} and by~\cite{russell2019efficient} assume that the model is linear. \cite{ribeiro2016should} and \cite{ribeiro2018anchors} provide explainability but they require a local approximation of the model and do not generate counterfactuals.
%\subsection{Fairness}
%\emph{Fairness.}
%Much of recent research on fairness of machine learning models has focused on developing metrics to measure whether or not a classifier is discriminating against people in groups defined by legally protected features (e.g., race, gender, age) and developing methods to mitigate the discrimination of methods with fairness issues.
% \jette{The following seems to general, but maybe would be good in the introduction?}
% In addition to these issues for machine learning models, counterfactuals pose two additional questions.
% First, if counterfactual explanations can be improved by changing features that are immutable (e.g., race), then what is the implication for the fairness of the model?
% Second, if counterfactuals being generated for groups delineated by their values of protected features markedly different from one another, then what does this imply about the fairness of the model.
% %\methodName~helps explore these questions and can be used by individuals and model developers to audit the model.
% \jette{End my general thoughts}
%Using counterfactual explanations to assess the fairness of machine learning models has not been explored in depth. %in regard to the fairness of counterfactuals and what counterfactuals imply about the fairness of the original model.
%\cite{wachter2017counterfactual} and \cite{russell2019efficient} posit using counterfactual explanations to measure the fairness or bias of model, but do not present a concrete method for analysis.
%For one dataset, \cite{ustun2019actionable} compare the costs of counterfactual explanations (i.e., some measure of how easily the counterfactual explanation could be achieved) for individuals that belong to groups defined by legally protected attributes (e.g., race, gender).
% Since~\methodName~is a post-hoc process it falls more in line with assessing the fairness of machine learning models.
% Several metrics have been put forth to decide if a machine learning model is fair.
% Disparate impact for a binary classifier, which is motivated by legal definition of discrimination, is the ratio between the probability that a person in a protected class receives a favorable outcome from a classifier and the probability of a person not in the protected class receives a favorable prediction from a classifier.
% If this ratio is less than 80\%, then the algorithm is said to be unfair.
% While motivated by legal restrictions, disparate impact does not take into account the actual label
%\subsection{Robustness}
%\emph{Robustness.}
%A lot of work has been done to generate adversarial examples and develop techniques to combat attacks, especially for image recognition tasks. \cite{carlini2017towards} used a similar formulation as ours to generate closest images that change the prediction by solving an optimization problem. However they use the L$_p$ norm to find the closest image, which is not a good measure of perceptual similarity.
%A more suitable metric is SSIM (Structural Similarity Index Measure) which is a better measure of image similarity from a humans perspective, as shown in \cite{wang2003multiscale}. Moreover, they conduct experiments on simpler convolution networks and do not define a measure for robustness. \cite{weng2018evaluating} define a robustness score using an extreme value theory approach. This is not a black-box approach though and requires the computation of gradients. Other GAN based approaches to generate adverarial attacks include \cite{xiao2018generating},\cite{hu2017generating} and \cite{song2018generative} where the goal is to find images close to an input, however these require extensive training while \methodName~ does not need to train any machine learning model.
\section{The CERTIFAI framework}
In this section, we formulate a custom genetic algorithm to find counterfactual(s).
Consider a black-box classifier \textit{f} and an input instance \textbf{x}. Let the counterfactual be a feasible generated point \textbf{c}.
Then the problem can be formulated as:
\begin{equation}
\begin{aligned}
\min_{c} d(\textbf x,\textbf c)\\
\textrm{s.t.} f(\textbf c)\neq f(\textbf x) \\
\end{aligned}
\label{eq:opt}
\end{equation}
\noindent where $d(\textbf x,\textbf c)$ is the distance between $\boldsymbol{x}$ and $\boldsymbol{c}$. To avoid using any approximations to or assumptions for the model, we use a genetic algorithm to solve Equation~\ref{eq:opt}.
The custom genetic algorithm works for any black-box model and input data type, and it is model-agnostic.
Additionally, it provides a great deal of flexibility in counterfactual generation.% to both the user and model creator.
\methodName's genetic algorithm solves the optimization problem in Equation~\ref{eq:opt} through a process of natural selection.
The only mandatory inputs for the genetic algorithm are the black-box classifier $f$ and an input instance \textbf{x}.
%For a given population size, the algorithm first generates a random set of individuals, $I_q$.
Generally, for an $n$-dimensional input vector \textbf{x}, let $W\in \mathbb{R}^{n}$ represent the space from which individuals can be generated and $P$ be the set of points with the same prediction as \textbf{x}:
\begin{equation}
P = \{\textbf{p} |f(\textbf{p}) = f(\textbf{x}), \textbf{p} \in W\}.
\end{equation}
The possible set of individuals $\textbf{c}\in I$ are defined such that
\begin{equation}
I = W\setminus P.\
\label{eq:I}
\end{equation}
Each individual $\textbf{c}\in I$ is a candidate counterfactual.
The goal is to find the fittest possible $\boldsymbol{c^*}$ to \textbf{x} constrained on $\boldsymbol{c^*} \in I$.
The fitness for an individual \textbf{c} is defined as:
\begin{equation}
\begin{aligned}
fitness = \frac{1}{d(\textbf x, \textbf c)} .\\
\end{aligned}
\label{eq:fit}
\end{equation}
Here $\boldsymbol{c^*}$ will then be the point closest to \textbf{x} such that $\boldsymbol{c^*} \in I$.
For a multi-class case, if a user wants the counterfactual \textbf{c} to be belong to a particular class $j$, we define $Q$ as:
\begin{equation}
Q = \{\textbf{q} |f(\textbf{q}) = j, \textbf{q} \in W\} .
\end{equation}
Then Equation \ref{eq:I} becomes:
\begin{equation}
I = (W\setminus P) \cap Q.
\label{eq:Imod}
\end{equation}
The algorithm is carried out as follows:
first, a set $I_c$ is built by randomly generating points such that they belong to $I$.
Individuals $\boldsymbol{c}\in I_c$ are then evolved through three processes: selection, mutation, and crossovers.
Selection chooses individuals that have the best fitness scores (Equation~\ref{eq:fit}).
A proportion of these individuals (dependent on $p_m$, the probability of mutation) are then subjected to mutation, which involves arbitrarily changing some feature values.
A proportion of individuals (dependent on $p_c$, the probability of crossover) are then subjected to crossover, which involves randomly interchanging some feature values between individuals.
The population is then restricted to the individuals that meet the required constraint (Equation~\ref{eq:I} or Equation~\ref{eq:Imod}), and the fitness scores of the new individuals are calculated.
This is repeated until the maximum number of generations is reached.
Finally, the individual(s) $\boldsymbol{c^*}$ with the best fitness score(s) is/are chosen as the desired counterfactual(s)\footnote{ p$_m$=0.2 and p$_c$=0.5, which is standard in literature. The population size is the square of the input feature size with a maximum cap of 30,000. Grid-search is used to find the number of generations}.
% The genetic algorithm is a meta-heuristic evolutionary algorithm used to solve optimization problems using a process of natural selection.
% The only mandatory inputs for the genetic algorithm are the black-box model and the instance.
% For a given population size, the algorithm first generates a random set of individuals.
% Specifically, for an $n$-dimensional input vector, let $W\in \mathbb{R}^{n}$ represent the space from which individuals can be sampled and $P$ be the set of points with the same prediction as \textbf{x}:
% \begin{equation}
% P = \{\textbf{p} |f(\textbf{p}) = f(\textbf{x}), \textbf{p} \in W\}
% \end{equation}
% After generating the set $W$, we evaluate each point in $W$ using $f,$ which allows us to identify $P$.
% % The evolutionary process starts from a population of randomly generated individuals, and is an iterative process, with the population in each iteration called a generation.
% % In each generation, the fitness of every individual in the population is evaluated. The fitter individuals are stochastically selected from the current population, and each individual's genome (the set of feature values of an individual) is modified (recombined and randomly mutated) to form a new generation.
% % The new generation of candidate solutions is then used in the next iteration of the algorithm, until the maximum number of generations are crossed or the desired fitness value is achieved.
% The possible set of individuals $\textbf{i}\in I$ are generated and selected are such that:
% \begin{equation}
% I \subseteq W\setminus P\
% \end{equation}
% Each individual $\textbf{i}\in I$ is a candidate counterfactual. The closest possible \textbf{i} to \textbf{x} constrained on $\textbf{i}\in I$ is found by measuring the fitness of every individual and evolving the population to improve that fitness. The fitness function is defined as
% \begin{equation}
% \begin{aligned}
% fitness = \frac{1}{d(\textbf x, \textbf c)} \\
% \end{aligned}
% \end{equation}
% %Hence, an individual closer to the input point is fitter than one further away. Since \textbf{c} is always on the other side of the decision boundary, we do not need to worry about d(\textbf x, \textbf c) being 0.
% % \begin{algorithm}[tb]
% % \caption{Genetic Algorithm for Counterfactual}
% % \label{alg:algorithm}
% % \textbf{Inputs required}: Data point \textbf{x = [x$_1$, x$_2$, x$_3$,.... x$_j$]} , Model \textit{f}\\
% % \textbf{Optional Inputs}: data type of x$_i$'s, range of x$_i$'s, number of counterfactuals desired: k\\
% % \textbf{Parameters}: Number of generations g, Population size po, Probability of mutation p$_m$, Probability of crossover p$_c$\\
% % \textbf{Output}: Counterfactual \textbf{c}
% % \begin{algorithmic}[1] %[1] enables line numbers
% % \STATE P$_0$ = \{\textbf{p$_i$} $|$ \textit{f}(\textbf{p$_i$})!=\textit{f}(\textbf{x}) for all i = \{1,2,..po\}\}
% % \STATE popfitness = fitness(P$_0$,x)
% % \WHILE{i$<$g}
% % \STATE P$_{i+1}$ = selection(P$_0$)
% % \STATE Pm$_{i+1}$ = mutation(P$_{i+1}$,p$_m$)
% % \STATE Pc$_{i+1}$ = crossover(Pm$_{i+1}$,p$_c$)
% % \STATE P$_{i+1}$ = evaluate(Pc$_{i+1}$,\textbf{x},f)
% % \STATE popfitness = fitness(p$_{i+1}$,\textbf{x})
% % \STATE i=i+1
% % \ENDWHILE
% % \STATE j = index(max(popfitness))
% % \STATE \textbf{return} P$_g$[j]
% % \end{algorithmic}
% % \end{algorithm}
% The only mandatory inputs for the genetic algorithm are the black-box model and the instance. For a given population size, the algorithm generates a random set of individuals that satisfy (3), which are then evolved through three processes: selection, mutation and crossovers. Selection chooses individuals that have the best fitness scores (4). A proportion of these individuals (dependent on p$_m$, the probability of mutation) are then subjected to mutation, which involves arbitrarily changing some feature values. A proportion of individuals (dependent on p$_c$, the probability of crossover) are then subjected to crossover, which involves randomly interchanging some feature values between individuals. The population is then evaluated to ensure that the required constraints are satisfied and all the final individuals' fitness scores are calculated. This is repeated till the maximum number of generations allowed. Finally, the individual with the best fitness score is chosen as the desired counterfactual\footnote{ p$_m$ and p$_c$ are chosen as 0.2 and 0.5, which is standard in literature. The population size is the square of the input feature size with a maximum cap of 30,000. Grid-search is used to find the number of generations}.
\subsection{Choice of distance function}
The choice of distance function used in Equation~\ref{eq:opt} depends on the details provided by the model creator and the type of data being considered.
If the data is tabular, \cite{wachter2017counterfactual} demonstrated how the $L_1$ norm normalized by the median absolute deviation (MAD) is better than using the $L_1$ or $L_2$ norm for counterfactual generation.
For tabular data, the $L_1$ norm for continuous features (NormAbs) and a simple matching distance for categorical features (SimpMat) are chosen as default. In the absence of training data, normalization using MAD is not possible.
However in model development and our experimenets where there is access to training data, normalization is possible. The distance metric used is:
\begin{equation}
\begin{aligned}
d(\textbf{x},\textbf{c}) = \frac{n_{con}}{n}\text{NormAbs}(\textbf{x},\textbf{c}) + \frac{n_{cat}}{n}\text{SimpMat}(\textbf{x},\textbf{c})\\
\end{aligned}
\end{equation}
\noindent where $n_{con}$ and $n_{cat}$ are the number of continuous and categorical features, respectively, and $n$ is the total number of features ($n_{con} + n_{cat} = n$).
For image data, the Euclidean distance and absolute distance between two images are not good measures of image similarity \cite{wang2002image}.
Hence, we use SSIM (Structural Similarity Index Measure) \cite{wang2003multiscale}, which has been shown to be a better measure of what humans consider to be similar images \cite{wang2002image}.
SSIM values lie between 0 and 1, where a higher SSIM value means that two images look more similar to each other. For the input image \textbf{x} and counterfactual image \textbf{c}, the distance is:
\begin{equation}
d(\textbf{x},\textbf{c}) = \frac{1}{\text{SSIM}(\textbf{x},\textbf{c})}.
\label{SSIMeq}
\end{equation}
\subsection {Improving counterfactuals with constraints}
Apart from the input instance and black-box model, additional inputs help the algorithm produce better results. Auxiliary constraints are incorporated by restricting the space defined by the set $W$: the space from which individuals can be generated, to ensure feasible solutions. For an $n$-dimensional input, let $W$ be the Cartesian product of the sets $W_1$,$W_2$,...,$W_n$.
%Data types for each feature can be specified (e.g. $W_i \in \mathbb{Z}$ or $W_i \in \mathbb{R}$) otherwise we assume the data type in the format of the given input (e.g. 1:integer, 1.0: float).
%To avoid infeasible explanations, the value of features can be constrained.
For continuous features, $W_i$ can be constrained as $W_i \in [W_{imin}, W_{imax}]$, and categorical features can be constrained as $W_i \in \{W_1, W_2,...,W_j\}$. However, certain variables might be immutable (e.g., race). In these cases, a feature $i$ for an input \textbf{x} can be muted by setting $W_{i}= x_{i}$.
An example of the benefits of such constraints would be when a user may not want an explanation of an income change from \$10,000 to \$900,000 if that is not possible, so $W_i \in [\$10000,\$15000]$ might be an appropriate constraint.
The number of counterfactuals $k$ can also be set.
%The algorithm can generate the $k$ best individuals instead of just one.
\methodName~ chooses the top $k$ individuals ($k=1$ as default) where different features have changed, so the end-user can get multiple diverse explanations of different kinds.
\subsection {Robustness}
\label{sec:theory_rob}
Machine learning models are prone to attacks and threats. For example, deep learning models have performed exceedingly well for image recognition tasks, but it has been widely shown \cite{carlini2017towards}, \cite{nguyen2015deep} that these networks are prone to adversarial attacks. Two images may look the same to a human, but when presented to a model, they can produce different outcomes. A counterfactual is a generated point close to an input that changes the prediction and is therefore an adversarial example.
%To evaluate networks for robustness, we define a new black-box model robustness score: CERScore.
Given two black-box models, if the counterfactuals across classes are farther away from the input instances on average for one network as compared to the other network, that network would be harder to fool. Since \methodName~directly gives a measure of distance d(\textbf{x},\textbf{c}), this can be used to define the robustness score for a classifier.
Using this distance, we introduce Counterfactual Explanation-based Robustness Score (CERScore), the first ever black-box model robustness score. Given a model, the CERScore is defined as the expected distance between the input instances and their corresponding counterfactuals:
\begin{equation}
\begin{aligned}
CERScore(model)= \mathop{\mathbb{E}}_{X}[d(\textbf{x},\textbf{c}^*)].
\end{aligned}
\label{eq:burd}
\end{equation}
%with the number of possible predicted classes $k,$ consider $n$ input instances per predicted class $k$ for which we generate counterfactuals $\{c^*\}$. Then,
To be able to better compare models trained on different data sets, the CERScore can be normalized by the expected value of the distance between data points in each class over all classes \textit{k}, and hence we get the normalized CERScore NCERScore (abbreviated as NC) as:
\begin{equation}
\begin{aligned}
NC=\frac{\mathop{\mathbb{E}}_{X}[d(\textbf{x},\textbf{c}^*)].}{\sum_{k=1}^{K} P(x\in \text{class}_k) \mathbb{E}[d(x_i,x_j);x_i,x_j \in \text{class}_k]}
\end{aligned}
\label{eq:cernorm}
\end{equation}
\noindent (i.e., we normalize by dividing by the expected distance between two datapoints drawn from the same class). A higher CERScore implies that the model is more robust. Note that the normalized CERScore can be greater than 1. Unlike \cite{weng2018evaluating}, \methodName~only needs model predictions and not the model internals.
\subsection{Fairness}
\label{sec:fair}
The fitness measure (Equation \ref{eq:fit}) and CERScore can also be used to investigate fairness from individual and group perspectives, respectively.
For a given individual instance, if the genetic algorithm can generate different counterfactuals with different values of a protected feature (e.g., race, age), and as a result the user can achieve the desired outcome more easily than when those features could not be changed, then the individual could claim the model is unfair to their case.
Additionally, \methodName~can be used by model developers to audit the fairness for different groups of observations.
If the fitness measure is markedly different for counterfactuals generated for the different partitions of a feature's domain value, this could be an indication the model is biased towards one of the partitions.
For example, if the gender feature is partitioned into two values (male and female), and the average fitness values of generated counterfactuals are lower for females than for males, this could be used as evidence that the model is not treating females fairly.
%We define this measure of extra effort that a group has to make on average to achieve a desired outcome from a model to be ``disparate burden."
Using counterfactuals and the distance function, we can calculate the overall burden for a group, measured as:
%\DeclareMathOperator{\E}{\mathbb{E}}
\begin{equation}
\begin{aligned}
Burden(g)= \mathop{\mathbb{E}}_{g}[d(\textbf{x},\textbf{c}^*)]
\end{aligned}
\label{eq:burd}
\end{equation}
where \emph{g} is a partition defined by the distinct values for a specified feature set.
Note, burden is related to CERScore as it is the expected value over a group.
Most fairness auditing models focus on single features (e.g., \cite{hardt2016equality,donini2018empirical}). Burden, however, does not have that limitation and can be applied to any combination of features.
% \begin{itemize}
% \item Tabular data: \cite{wachter2017counterfactual} demonstrated how the $L_1$ norm normalized by the median absolute deviation (MAD) is better than using the $L_1$ or $L_2$ norm for counterfactual generation. For tabular data, the $L_1$ norm for continuous features (\textxit{df}) and a simple matching distance for categorical features (\textxit{cf}) is chosen as default. In the absence of training data or the MAD value, normalization isn't possible however in our experiments, we do normalize and the metric used is:
% \begin{equation}
% \begin{aligned}
% d(\textbf{x},\textbf{c}) = \frac{df}{n}NormAbs(\textbf{x},\textbf{c}) + \frac{cf}{n}SimpMat(\textbf{x},\textbf{c})\\
% \end{aligned}
% \end{equation}
% Where n is the total number of features (\textit{df} + \textit{cf} = \textit{n}).
% \item Image data: We produce images that are as similar looking to the input image as possible (with a changed prediction). The euclidean distance or absolute distance between two images aren't good measures of image similarity and human perceptibility. Hence, we use SSIM (Structural Similarity index measure) \cite{wang2003multiscale} which has been shown to be a better measure of what humans consider to be similar images. SSIM values lie between 0 and 1, where a higher SSIM value means that two images look more similar to each other. For the input image \textbf{x} and counterfactual image \textbf{c}:
% \begin{equation}
% d(\textbf{x},\textbf{c}) = \frac{1}{SSIM(\textbf{x},\textbf{c})}
% \end{equation}
% %\begin{equation}
% %SSIM(x,c)=\frac{(2\mu_x\mu_c + C_1) + (2 \sigma _{xc} + C_2)}
% % {(\mu_c^2 + \mu_c^2+C_1) (\sigma_x^2 + \sigma_c^2+C_2)}
% % \label{eq:SSIM}
% %\end{equation}
% %Here $\mu_x $ and $ \mu_c $ are the average pixel values of images $x$ and $c,$ $\sigma_x$ and $ \sigma_c $ are the standard deviation of pixels.$ \sigma _{xc} $is the covariance between images x and c and $ C_1 $and $C_2 $ are constants to stabilize a weak denominator.
% \end{itemize}
\section{Experiments}
\begin{figure}[tp]
\centering
\includegraphics[width = 6cm, height = 4cm]{cour.PNG}
\caption{Adversarial examples for the MNIST dataset. The images on the left are original inputs for which the model has a correct prediction and the images on the right are counterfactual images (adversarial examples) for which the prediction has changed.}
\label{fig:adv_examples}
\end{figure}
We demonstrate the applications and flexibility of \methodName~ to explainability, transparency, fairness, and robustness.
\subsection{Robustness}
In this section, we demonstrate how \methodName~ produces adversarial examples, and we use CERScore from Section~\ref{sec:theory_rob} to measure a network's resistance to adversarial attacks.
\subsubsection{Generating Adersarial Examples}
We consider the MNIST dataset \cite{lecun1998mnist} which contains 60000 (size 28x28) training images of digits. We use it to train a convolutional neural network, consisting of one convolution layer, two dense layers, and intermediate pooling and dropout layers. We achieve a 99.46\% accuracy on the test set using this architecture. Then we select random images from the MNIST dataset and the model above and find the counterfactual image for each image, using SSIM (equation (8)). Every pixel is considered to be a feature and hence, every individual in the population is a 784 dimension vector. We use an initial population size of 30,000 individuals and run the experiment for 1,000 generations.
\begin{table}
\centering
\begin{tabular}{lrrrr}
\toprule
Model & CERScore & CI &CLEVER\\
\midrule
Inception-v3 & 1.17 & 1.09-1.25 &0.229 \\
Resnet-50 & 1.06 & 1.05-1.08 &0.137 \\
MobileNet & 1.08 & 1.06-1.09 &0.151 \\
\bottomrule
\end{tabular}
\label{tab:CERScore}
\caption{Robustness score and 95 percent confidence intervals (CI) for those scores for 3 deep learning models and the corresponding CLEVER scores} %\cite{weng2018evaluating}}
\end{table}
\begin{table*}[]
\centering
\begin{tabular}{lcccccccc}
\toprule
\textbf{Data set} & \textbf{Num.} & \textbf{Num.} & \multicolumn{2}{c}{\textbf{DT}} & \multicolumn{2}{c}{\textbf{SVM}} & \multicolumn{2}{c}{\textbf{MLP}} \\
& \textbf{obs.} & \multicolumn{1}{l}{\textbf{features} } & \textbf{NCERS.} & \textbf{Acc.} & \textbf{NCERS.} & \textbf{Acc.} & \textbf{NCERS.} & \textbf{Acc.} \\
\midrule
Pima Diabetes & 768 & 8 & 0.074 & 73.25 & 0.387 & 81.42 & \textbf{0.486} & 98.61 \\
Breast Cancer & 569 & 32 & 0.081 & 95.80 & \textbf{0.121} & 96.50 & \textbf{0.124} & 96.50 \\
Iris \small{ } & 150 & 4 & 0.132 & 95.67 & 0.235 & 95.67 & \textbf{0.241} & 95.67 \\ \bottomrule
\end{tabular}
\label{tab:data}
\caption{Descriptions of data sets, and NCERScore (NCERS.) and test set accuracy (Acc.) for three models: decision tree (DT), SVM with RBF kernel (SVM), and Multilayer Perceptron (MLP).}
\end{table*}
Figure~\ref{fig:adv_examples} shows an example of ten generated counterfactuals (right) and their original counterpart (left) images.
The counterfactual images on the right look nearly identical to the input images on the left, however, the model predicts a different outcome for the images on the left and right. The imperceptibly different images give credence to the idea of using a genetic algorithm formulation to produce counterfactuals. Additionally, our approach towards generating these images is model-agnostic and does not require any approximations, unlike \cite{carlini2017towards}. The generated images show how a network can easily be fooled and demonstrate that there is a major problem in deploying such highly-accurate networks to image-based decision making applications (eg. face recognition). Moreover, different kinds of adversarial attacks can be generated by simply changing the distance function in Equation \ref{eq:fit}.
%\vspace{-0.07in}
\subsubsection{Evaluating Deep Networks}
% To evaluate networks for robustness, we define a new black-box model robustness score: CERScore. To evaluate if it accurately describes robustness, we compute the corresponding CLEVER scores \cite{weng2018evaluating}.
% As demonstrated in figure \ref{fig:adv_examples}, the counterfactual images look very similar to original images. Given two models, if these counterfactual images are further away (different looking) from the input images for one network as compared to the other network on average, that network would be harder to fool. Since our method directly gives us a measure of distance d(\textbf{x},\textbf{c}), where \textbf{x} in this case is the input image and \textbf{c} is the counterfactual, this can be used to define the robustness score for a network.
% We propose the first ever black-box model robustness score CERScore: Counterfactual based Robustness Score. Given a model with the number of possible predicted classes k, consider n images per predicted class k for which we generate counterfactuals (total number = $n\cdot k$). Then,
% \begin{equation}
% \begin{aligned}
% CERScore= \frac{1}{nk}\sum_{i=1}^k \sum_{j=1}^n d(\textbf{x$_i$$_j$},\textbf{c$_i$$_j$})
% \end{aligned}
% \end{equation}
% Where $\textbf{x}_{ij}$ is the $j^{th}$ input image belonging to class $i$, and $\textbf{c}_{ij}$ is the corresponding counterfactual. CERScore is \textbf{hence} the average of the distance between the counterfactual and original image for $n$ images each across $k$ classes. Hence, a higher CERScore implies that the model is more robust. Unlike \cite{weng2018evaluating}, we do not need the model weights or gradients, just the prediction.
In this section, we evaluate how well CERScore, introduced in Section~\ref{sec:theory_rob}, can give an informative measure of robustness.
We consider the same networks as in \cite{weng2018evaluating}: Inception-v3 \cite{szegedy2016rethinking} , ResNet-50 \cite{he2016deep} and MobileNet \cite{howard2017mobilenets} pre-trained on ImageNet \cite{deng2009imagenet}, where they define the CLEVER score for robustness. Unlike CLEVER, we consider the model to be a black-box (only relying on its predictions). Ideally, to derive a measure of robustness for a model, all images from all classes should be considered, their counterfactuals should be generated, and the CERScore should then be calculated. However, since the number of training samples for a deep network is in the order of millions, it is not computationally feasible to calculate the score for each example. Hence, we consider a subset of classes and images to calculate the CERScore. We sampled \emph{n}=50 random images from every class across \emph{k}=100 random classes. We generate the counterfactuals for all 5,000 images such that the counterfactual gives a prediction of the second most likely class (by generating individuals constrained on belonging to that class as in Equation~\ref{eq:Imod}) and empirically estimate the CERScore as:
\begin{equation}
\begin{aligned}
CERScore= \frac{1}{nk}\sum_{i=1}^k \sum_{j=1}^n d(\textbf{x$_i$$_j$},\textbf{c$^*_{ij}$})
\end{aligned}
\end{equation}
%\begin{equation}
%\begin{aligned}
%CERScore= \frac{1}{nk}\sum_{i=1}^k \sum_{j=1}^n d(\textbf{x$_i$$_j$},\textbf{c^*$_i$$_j$})
%\end{aligned}
%\end{equation}
\noindent where $\textbf{x}_{ij}$ is the $j^{th}$ input instance belonging to predicted class $i$, and $\textbf{c}^*_{ij}$ is the corresponding counterfactual. The CERScores are shown in Table~\ref{tab:CERScore}. One way to interpret the score is that on average, the SSIM score for Inception-v3 is 1/1.17 = 0.85, where an SSIM score of 1 means the images look exactly the same and an SSIM score of 0 means the images are highly different. Hence, adversarial attacks for Inception-v3 could be more easily identified than for the other models.
We also show the 95\% confidence interval where we have assumed the distribution of distances between the images and their counterfactuals follows a normal distribution.
The confidence intervals are tight around the CERScores.
We also compare CERScore with CLEVER scores \cite{weng2018evaluating} for the same images, considering the top-2 class attack. The CLEVER scores are also reported in Table~\ref{tab:CERScore}. The CERScore implies that Inception-v3 is most robust and Resnet-50 is least robust, which is similar to what the CLEVER scores suggest. Hence, even though \methodName~ does not access any model weights, it is able to evaluate a model's robustness to adversarial attacks.
\subsubsection{Robustness of Classic Classifiers}
Next, we use NCERScore (Equation \ref{eq:cernorm}) to compare the robustness of different models trained on different data sets.
We train three models (decision trees (DT), Support Vector Machines with RBF kernel (SVM), and multilayer perceptrons (MLP)) on the three data sets listed in Table~\ref{tab:data}.
We report the NCERScore and the accuracy on the test set in Table~\ref{tab:data}.
%data sets: the PIMA diabetes data set, the Iris Flowers data set and the Wisconsin Breast Cancer data set, details of which are given in \ref{tab:data}. We use 3 models: Decision Trees (DT), Support Vector Machines (SVM) and multilayer perceptrons (MLP), and train the three models on each of the three data sets. Thereafter, we find the NCERScore for each model using Equation \ref{eq:cernorm}. The test accuracy for each model along with the NCERScore are shown in Table \ref{tab:data}.
%All normalized NCERScore values are lower than 1, reflecting that the counterfactuals generated on average are close to the input points compared to the expected value of the distance between points per class in the data set. \jette{is this always true? I think we can take this out?}
Across all data sets, the neural network has the highest NCERScore and is therefore the most robust of the classifiers for these data sets.
In the Pima diabetes data set, the accuracy of the decision tree is much lower than the other models, which suggests this simple model cannot adequately capture the class separation.
Hence, more points would be concentrated near the decision boundaries, resulting in a lower NCERScore.
\begin{figure*}[tp]
\centering
\includegraphics[width = 14cm, height = 5cm]{Cermod1.png}
\caption{Decision boundaries and input points for the Iris flowers data set (3 classes) for 3 models: Neural Network, SVM with RBF kernel and Decision tree (DT), visualized using two features from the data set. DT has closer points to the decision boundaries on average.}
\label{fig:DT}
\end{figure*}
%Rows two and three demonstrate how the NCERScore is actually a good measure of robustness.
For the Iris data set, while it is a relatively simple data set (even the decision tree performs well), the decision tree has the lowest NCERScore while the scores for SVM and MLP are similar.
In Figure~\ref{fig:DT}, we plot input points for two features of the Iris data set and the decision boundary for each model.
Looking closely, the points for the decision tree are closer on average to the decision boundaries as compared to the other two models (i.e. the densely clustered white and red points are closer to decision boundaries), which suggests the model is more prone to being fooled.
The decision boundaries for both SVM and MLP are nearly identical around the input points, which results in similar robustness scores.
Similar results can be seen for the cancer data set. Using these results, a model developer can choose the most robust model based on the NCERScore.
\begin{table}
\centering
\begin{tabular}{lrrrr}
\toprule
Person & Feature(s) & Original & Counterfactual\\
\midrule
1 & Glucose (CWC) & 115 & 71 \\
&BMI (CUC) & 35.3 & 10.1 \\
\midrule
2 & Glucose (CWC) & 168 & 89 \\
&Age (CUC) & 34 & 44 \\
\bottomrule
\label{tab:PIMA}
\end{tabular}
\caption{Counterfactual explanations for the Pima Indian diabetes dataset. CWC: counterfactuals with constraints on feature values and CUC: unconstrained counterfactuals. Unconstrained features lead to infeasible solutions (BMI 10.1) or unchangeable features (age) being changed.}
\label{tab:Hi}
\end{table}
\subsection{Explainability}
\begin{table}
\centering
\begin{tabular}{lrrrr}
\toprule
Person & Feature(s) & Original & Counterfactual\\
\midrule
1 & Education & 12th & Bachelors \\
& Occupation & Tech-suppt & Exec-managerial \\
\midrule
1 & Hrs-per-week & 50 & 70 \\
&Workclass & Local-gov & Private \\
\bottomrule
\label{tab:UCI}
\end{tabular}
\caption{Two explanations for the same person from the UCI adult dataset, with constraints on feature values.}
\label{tab:boo}
\end{table}
Counterfactuals are used to provide explanations and transparency to a user on how much change is needed for them to obtain a favorable prediction. We show the importance of using the constraints to improve explanations, the use of multiple counterfactual explanations for a single instance, and how these can also be used to estimate feature importance.
\subsubsection{Datasets and Models}
We use the Pima Indian dataset \cite{smith1988using} and the UCI adult dataset \cite{kohavi1996scaling} in the following experiments.
The Pima Indian dataset consists of 768 data samples and 8 features where 6 features are continuous and integer-valued, and 2 features are continuous float-valued. The task is to predict the risk of diabetes (1: At risk, 0:Not at risk). We train a 4 layer neural network with an input layer, 2 hidden layers of 20 neurons each, and an output layer with a 80-20 training-test split. The accuracy of the model is 99.6\% on the test set. An initial population of 500 individuals is considered and the evaluation is done across 300 generations.
The UCI adult dataset consists of 48842 samples with 14 categorical and continuous integer features and a binary outcome of predicted income ($>$50k or $<=$50k). Since the dataset contains many categorical variables, finding a counterfactual using \cite{wachter2017counterfactual} would not be feasible. We train a 6 layer neural network with an input layer, 4 hidden layers of 80 neurons each, and an output layer with a 80-20 training-test split. The accuracy of the model is 99.20\% on the test set. Since the dataset is larger, an initial random population of 1000 individuals is considered and the evaluation is done across 500 generations. The negative outcome is considered to be income $<=$50k, and we find the counterfactuals for those.
We only consider those input instances where the model prediction matches the ground-truth.
\begin{figure}[tp]
\includegraphics[width = 8.5cm,height = 5.3cm]{FTIMPF.png}
\caption{Feature importance for the model, trained on the Pima Indian diabetes dataset, measured by the number of times a feature changed to generate the counterfactual (left) and feature importance by XGBoost (right).}
\label{fig:Pima Indian_feat_imp}
\end{figure}
\subsubsection{Importance of Constraints}
We consider two cases of counterfactual generation, counterfactuals with constraints (CWC) and counterfactuals unconstrained (CUC) for users with a prediction of high diabetes risk. CWC corresponds to a user or model creator providing a range of values for features. CUC corresponds to a user only providing the black-box model and the input instance without any constraints on the feature values. We show features for which the values have changed (between the input and counterfactual), all other values remained constant.
As shown in Table~\ref{tab:Hi}, for person 1, when we provide constraints (CWC), the explanation is: \textit{Had your glucose been less by 34, you wouldn't have been at the risk of diabetes}. All other feature values for the user remained constant. Without constraints, the explanation shows that the BMI would have to be decreased to 10.1. While this is a smaller change in magnitude as compared to changing the glucose level, achieving a BMI of 10.1 is not feasible, and hence it is important to use the flexibility of our approach to add additional constraints that ensure feasibility. Similarly, for person 2, the age is suggested to be changed, which is not feasible.
\subsubsection{Measuring feature importance}
From a model developer's perspective, counterfactuals can show the importance of every feature value to the prediction and hence provide transparency. If \methodName~is changing a particular feature more often than another feature when comparing the input and counterfactual, that feature is more significant for a model. For the Pima Indian diabetes dataset, we generate counterfactuals for all samples (irrespective of prediction) and analyze the number of times every feature value has changed, as shown in Figure~\ref{fig:Pima Indian_feat_imp}. Interestingly, the importances are qualitatively similar to those returned by Python's XGBoost \cite{Chen:2016:XST:2939672.2939785} library (also shown in Figure~\ref{fig:Pima Indian_feat_imp}). Specifically, feature 5 (BMI) and feature 2 (Glucose) are the most important in predicting diabetes risk. This analysis can be extended to the multi-class case by constraining sampled individuals such that they belong to a desired class (Equation~\ref{eq:Imod})
\begin{table}
\centering
\begin{tabular}{lrrrr}
\toprule
Person & Feature & FitnessM & FitnessU\\
\midrule
1 & Race & 0.63 & 0.87 \\
\midrule
2 & Gender & 0.41 & 0.62 \\
\midrule
3 & Race & 0.81 & 0.81 \\
\bottomrule
\end{tabular}
\caption{Fitness values when race and gender attributes are muted (FitnessM) and unmuted (FitnessU) for three people.}
\label{tab:ind_fairness}
\end{table}
\subsubsection{Multiple counterfactual explanations}
Multiple explanations are helpful to a user so that they can receive a diverse set of changes that could be made to achieve a desired outcome. The UCI adult dataset (CWC case) is considered and features such as native-country are muted and a set range is given for features like hours-per-week. We run the genetic algorithm for the input instance and select the best two individuals that have different changes in feature indices. The advantage of our approach is that we only need to run the algorithm once, and we can generate many explanations, as opposed to \cite{russell2019efficient} where the IP solver needs to be run multiple times to generate multiple explanations. %Other approaches like \cite{wachter2017counterfactual} generate single explanations rather than a set of diverse choices.
To underscore the benefits of suggesting alternative counterfactuals, Table~\ref{tab:boo} shows two sets of explanations that are generated by \methodName~for the same person.
%: \textit{Had your education been Bachelors and your occupation been Exec-managerial or Had your hours per week been 70 and your workclass been private, your income would be $>$50k}.
Multiple explanations, the number of which is set by the user, allow a user to decide which counterfactual may be the most actionable.
%In some cases when the constraints specified are not viable for a desired prediction, the algorithm generates an empty set and hence the user can make no possible change to get a favorable prediction. For example, for the UCI adult dataset, for a particular example case, when the user chose their education, occupation and workclass as immutable and provided a range for capital gain and capital loss, no counterfactual was generated. When this occurs, we generate an explanation stating: \textit{No possible change for the constraints provided}
\subsection{Fairness}
We evaluate fairness from an individual's perspective and from a model developer's perspective.
To see if the model is unfair towards any instance, we consider 100 random instances of the UCI adult dataset where the prediction was unfavorable and run the algorithm twice, once when the sensitive attribute is not allowed to change and once when it is, and record the fitness values. We do this for two sensitive attributes, race and gender.
The results for three such instances are shown in Table~\ref{tab:ind_fairness}. FitnessM refers to the fitness value when the race feature is muted for an individual and FitnessU corresponds to the feature being unmuted. The fitness for the first 2 people increases substantially when these protected features are allowed to change and hence for these instances, there is evidence that the model has not been fair. For the third person, the evidence suggests that the model has been fair.
\begin{figure}[tp]
\centering
\includegraphics[width = 8.5cm, height = 6cm]{efg.png}
\caption{Burden on different groups belonging to a particular race in the UCI adult dataset, found using the distance between the input instances and counterfactuals (Equation~\ref{eq:burd})}
\label{fig:burden}
\end{figure}
%Disparate burden gives a sense of if a model is unfair to a certain group, and by how much. Since we measure the distance of each counterfactual to the corresponding input instance, the value can be used to estimate disparate burden.
A model developer can use the idea of burden (Equation~\ref{eq:burd}) to evaluate how fair a model is being to groups of individuals.
To demonstrate the idea of burden, we consider the attribute race in the UCI adult dataset and take all training examples that have an unfavorable outcome.
Results of our experiments are shown in Figure~\ref{fig:burden}. As we can see, the burden on Black race and the Other race is more than the other races. This means that on average, these groups would have to make more changes to achieve a desired prediction as compared to others. Hence the model imposes a burden on these groups, which could imply that the model has been unfair.
% We investigate fairness with respect to an individual instance (from a user perspective) and for the entire model, given the training set (from a model creators perspective). Such information is extremely useful to ensure that the model is not biased towards a particular group of individuals. For example, when race is allowed to be changed in the UCI adult dataset experiments, one explanation for an Asian-Pac-Islander user stated: \textit{Had your race been White, your income would be $>$50k}. Such an explanation shows how the model has been unfair to the user for this instance and causes legal implications.
% A direct consequence of our counterfactual generation approach is that we use the distance between the counterfactual or a set of counterfactuals and the input instance as a fitness measure, and hence this value can be used to gauge how fair a model is. Here, we define fairness with respect to an instance and with respect to the model:
% \begin{itemize}
% \item \textbf{Definition 1}: The model is unfair towards a given instance if allowing the change of a sensitive attribute produces a better counterfactual to that instance.
% \item \textbf{Definition 2}: Disparate burden is defined as the additional cost that the model imposes on a particular group to achieve a favorable outcome.
% \end{itemize}
% Definition 1 implies that if a feature like race or gender is allowed to be changed and the user can achieve a more favorable outcome easily i.e. the fitness of the best individual generated by the genetic algorithm improves, then the model has been unfair to that instance. To see if the model is unfair towards any instance, we consider 100 random instances of the UCI adult dataset where the predicition was unfavorable and run the algorithm twice, once with the sensitive attribute muted and once with it unmuted, and record the fitness values. We do this for 2 sensitive attributes, race and gender.
% The results for 3 such instances are shown in table 3. FitnessM refers to the fitness value when the race feature is muted for an individual and FitnessU corresponds to the feature being unmuted. The fitness for the first 2 people increases when these features are allowed to be changed and hence for these instances (persons), the model has not been fair. For the third person, the fitness remains the same so the model has been fair.
% Disparate burden is a measure of the extra effort that a group has to make on average to achieve a desired outcome from a model. Hence, it gives a sense of if a model is unfair to a certain group, and by how much. Since we measure the distance of each counterfactual to the corresponding input instance, the value can be used to estimate disparate burden. We consider the attribute race in the UCI adult dataset and take all training examples that have an unfavorable outcome. Counterfactuals with constraints (race itself is muted) are generated and the overall burden for a group is measured as:
% %\DeclareMathOperator{\E}{\mathbb{E}}
% \begin{equation}
% \begin{aligned}
% Burden(g)= $\displaystyle \mathop{\mathbb{E}}_{race=g}$[d(\textbf{x},\textbf{c})]
% \end{aligned}
% \end{equation}% Where g refers to the particular group for which we calculate the burden. Results of our experiments are shown in figure 4. As we can see, the burden on Black race and the Other race is more than the other races. This means that on average, these races would have to make more changes to achieve a desired prediction as compared to other races. Hence the model imposes a burden on these groups. To combat this, the burden value above can be provided to a model creator to re-train the model, taking this into account.
\section{Conclusion and Future Work}
In this paper, we introduced \methodName, a model-agnostic, flexible, and user-friendly technique that helps build responsible artificial intelligence systems. We demonstrate the flexibility that the genetic algorithm brings to provide feasible counterfacual explanations to a user. We show how individual and group fairness can be measured using the fitness values obtained during counterfactual generation. Finally, we show how these counterfactuals are effective adversarial examples and we define CERScore, the first ever measure of robustness for a black-box model.
We are currently developing the User-Interface to CERTIFAI. Future work involves speeding up the genetic algorithm by techniques like \cite{harik1999compact} and \cite{mitchell1994will} %and using image segmentation to group of pixels as features so that the CERScore is generated in real-time for images.
A comparison between the introduced fairness metric and previous metrics would also be useful. It would also be interesting to see how our adversarial examples perform with strategies \cite{papernot2016distillation}, \cite{madry2017towards} that are aimed to handle adversarial attacks.
%% The file named.bst is a bibliography style file for BibTeX 0.99c
\bibliographystyle{named}
\bibliography{ijcai19}
\end{document}
\begin{thebibliography}{}
\bibitem[\protect\citeauthoryear{Akhtar and Mian}{2018}]{akhtar2018threat}
Naveed Akhtar and Ajmal Mian.
\newblock Threat of adversarial attacks on deep learning in computer vision: A
survey.
\newblock {\em IEEE Access}, 6:14410--14430, 2018.
\bibitem[\protect\citeauthoryear{Binns}{2017}]{binns2017fairness}
Reuben Binns.
\newblock Fairness in machine learning: Lessons from political philosophy.
\newblock {\em arXiv preprint arXiv:1712.03586}, 2017.
\bibitem[\protect\citeauthoryear{Carlini and Wagner}{2017}]{carlini2017towards}
Nicholas Carlini and David Wagner.
\newblock Towards evaluating the robustness of neural networks, 2017.
\bibitem[\protect\citeauthoryear{Chen and
Guestrin}{2016}]{Chen:2016:XST:2939672.2939785}
Tianqi Chen and Carlos Guestrin.
\newblock {XGBoost}: A scalable tree boosting system, 2016.
\bibitem[\protect\citeauthoryear{Deng \bgroup \em et al.\egroup
}{2009}]{deng2009imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li~Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock 2009.
\bibitem[\protect\citeauthoryear{Donini \bgroup \em et al.\egroup
}{2018}]{donini2018empirical}
Michele Donini, Luca Oneto, Shai Ben-David, John~S Shawe-Taylor, and
Massimiliano Pontil.
\newblock Empirical risk minimization under fairness constraints, 2018.
\bibitem[\protect\citeauthoryear{Guidotti \bgroup \em et al.\egroup
}{2018a}]{guidotti2018local}
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco
Turini, and Fosca Giannotti.
\newblock Local rule-based explanations of black box decision systems.
\newblock {\em arXiv preprint arXiv:1805.10820}, 2018.
\bibitem[\protect\citeauthoryear{Guidotti \bgroup \em et al.\egroup
}{2018b}]{guidotti2018survey}
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca
Giannotti, and Dino Pedreschi.
\newblock A survey of methods for explaining black box models.
\newblock {\em ACM computing surveys (CSUR)}, 51(5):93, 2018.
\bibitem[\protect\citeauthoryear{Hardt \bgroup \em et al.\egroup
}{2016}]{hardt2016equality}
Moritz Hardt, Eric Price, Nati Srebro, et~al.
\newblock Equality of opportunity in supervised learning, 2016.
\bibitem[\protect\citeauthoryear{Harik \bgroup \em et al.\egroup
}{1999}]{harik1999compact}
Georges~R Harik, Fernando~G Lobo, and David~E Goldberg.
\newblock The compact genetic algorithm.
\newblock {\em IEEE transactions on evolutionary computation}, 3(4):287--297,
1999.
\bibitem[\protect\citeauthoryear{He \bgroup \em et al.\egroup
}{2016}]{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition, 2016.
\bibitem[\protect\citeauthoryear{Howard \bgroup \em et al.\egroup
}{2017}]{howard2017mobilenets}
Andrew~G Howard, Menglong Zhu, Bo~Chen, Dmitry Kalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, and Hartwig Adam.
\newblock Mobilenets: Efficient convolutional neural networks for mobile vision
applications.
\newblock {\em arXiv preprint arXiv:1704.04861}, 2017.
\bibitem[\protect\citeauthoryear{Kohavi}{1996}]{kohavi1996scaling}
Ron Kohavi.
\newblock Scaling up the accuracy of naive-bayes classifiers: A decision-tree
hybrid., 1996.
\bibitem[\protect\citeauthoryear{LeCun}{1998}]{lecun1998mnist}
Yann LeCun.
\newblock The mnist database of handwritten digits.
\newblock {\em http://yann. lecun. com/exdb/mnist/}, 1998.
\bibitem[\protect\citeauthoryear{Madry \bgroup \em et al.\egroup
}{2017}]{madry2017towards}
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and
Adrian Vladu.
\newblock Towards deep learning models resistant to adversarial attacks.
\newblock {\em arXiv preprint arXiv:1706.06083}, 2017.
\bibitem[\protect\citeauthoryear{Mitchell \bgroup \em et al.\egroup
}{1994}]{mitchell1994will}
Melanie Mitchell, John~H Holland, and Stephanie Forrest.
\newblock When will a genetic algorithm outperform hill climbing, 1994.
\bibitem[\protect\citeauthoryear{Nguyen \bgroup \em et al.\egroup
}{2015}]{nguyen2015deep}
Anh Nguyen, Jason Yosinski, and Jeff Clune.
\newblock Deep neural networks are easily fooled: High confidence predictions
for unrecognizable images, 2015.
\bibitem[\protect\citeauthoryear{Papernot \bgroup \em et al.\egroup
}{2016}]{papernot2016distillation}
Nicolas Papernot, Patrick McDaniel, Xi~Wu, Somesh Jha, and Ananthram Swami.
\newblock Distillation as a defense to adversarial perturbations against deep
neural networks, 2016.
\bibitem[\protect\citeauthoryear{Ribeiro \bgroup \em et al.\egroup
}{2016}]{ribeiro2016should}
Marco~Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.
\newblock Why should i trust you?: Explaining the predictions of any
classifier, 2016.
\bibitem[\protect\citeauthoryear{Russell}{2019}]{russell2019efficient}
Chris Russell.
\newblock Efficient search for diverse coherent explanations, 2019.
\bibitem[\protect\citeauthoryear{Smith \bgroup \em et al.\egroup
}{1988}]{smith1988using}
Jack~W Smith, JE~Everhart, WC~Dickson, WC~Knowler, and RS~Johannes.
\newblock Using the adap learning algorithm to forecast the onset of diabetes
mellitus, 1988.
\bibitem[\protect\citeauthoryear{Szegedy \bgroup \em et al.\egroup
}{2016}]{szegedy2016rethinking}
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Wojna.
\newblock Rethinking the inception architecture for computer vision, 2016.
\bibitem[\protect\citeauthoryear{Ustun \bgroup \em et al.\egroup
}{2019}]{ustun2019actionable}
Berk Ustun, Alexander Spangher, and Yang Liu.
\newblock Actionable recourse in linear classification, 2019.
\bibitem[\protect\citeauthoryear{Wachter \bgroup \em et al.\egroup
}{2017}]{wachter2017counterfactual}
Sandra Wachter, Brent Mittelstadt, and Chris Russell.
\newblock Counterfactual explanations without opening the black box: automated
decisions and the gdpr.
\newblock {\em Harvard Journal of Law \& Technology}, 31(2):2018, 2017.
\bibitem[\protect\citeauthoryear{Wang \bgroup \em et al.\egroup
}{2002}]{wang2002image}
Zhou Wang, Alan~C Bovik, and Ligang Lu.
\newblock Why is image quality assessment so difficult?, 2002.
\bibitem[\protect\citeauthoryear{Wang \bgroup \em et al.\egroup
}{2003}]{wang2003multiscale}
Zhou Wang, Eero~P Simoncelli, and Alan~C Bovik.
\newblock Multiscale structural similarity for image quality assessment, 2003.
\bibitem[\protect\citeauthoryear{Weng \bgroup \em et al.\egroup
}{2018}]{weng2018evaluating}
Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao,
Cho-Jui Hsieh, and Luca Daniel.
\newblock Evaluating the robustness of neural networks: An extreme value theory
approach.
\newblock {\em arXiv preprint arXiv:1801.10578}, 2018.
\end{thebibliography}