Uncovering Unique Concept Vectors through Latent Space Decomposition
Extracted source text · arXiv 2307.06913
\documentclass[10pt]{article}
% Any additional packages needed should be included after jmlr2e.
% Note that jmlr2e.sty includes epsfig, amssymb, natbib and graphicx,
% and defines many common macros, such as 'proof' and 'example'.
%
% It also sets the bibliographystyle to plainnat; for more information on
% natbib citation styles, see the natbib documentation, a copy of which
% is archived at http://www.jmlr.org/format/natbib.pdf
\input{math_commands.tex}
\usepackage[preprint]{tmlr}
\usepackage{hyperref}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{lineno}
\usepackage{amsmath}
\usepackage{amssymb}
%\usepackage{booktabs} % for professional tables
%\usepackage{mathrsfs}
%\usepackage{amsmath}
\usepackage{natbib}
\usepackage{amstext}
\usepackage{subcaption}
\usepackage{appendix}
\usepackage{subfiles}
%\usepackage{todonotes}
\usepackage{url}
%\linenumbers
\newcommand{\dataset}{{\cal D}}
\title{Uncovering Unique Concept Vectors through Latent Space Decomposition}
\author{Mara Graziani \email mara.graziani@hevs.ch\\
\addr Institute of Informatics\\
University of Applied Sciences of Western Switzerland (Hes-so Valais)\\
Sierre, CH 3960, Switzerland
\AND
\name Laura O' Mahony \email LauraA.OMahony@ul.ie\\
\addr Division of Computer Science and Department of Statistics\\
University of Limerick\\
Limerick, V94 T9PX, Ireland
\AND
\name An-Phi Nguyen \email an-phi.nguyen@biognosys.com\\
\addr Biognosys\\
Zurich, CH 8952, Switzerland
\AND
\name Henning Müller \email henning.mueller@hevs.ch \\
\addr Institute of Informatics\\
University of Applied Sciences of Western Switzerland (Hes-so Valais)\\
Sierre, CH 3960, Switzerland
\AND
\name Vincent Andrearczyk \email vincent.andrearczyk@hevs.ch \\
\addr Institute of Informatics\\
University of Applied Sciences of Western Switzerland (Hes-so Valais)\\
Sierre, CH 3960, Switzerland
}
%\editor{Editor Name}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\def\month{MM} % Insert correct month for camera-ready version
\def\year{YYYY} % Insert correct year for camera-ready version
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
\begin{document}
\maketitle
\begin{abstract}% <- trailing '%' for backward compatibility of .sty file
Interpreting the inner workings of deep learning models is crucial for establishing trust and ensuring model safety.
Concept-based explanations have emerged as a superior approach that is more interpretable than feature attribution estimates such as pixel saliency.
However, defining the concepts for the interpretability analysis biases the explanations by the user's expectations on the concepts.
To address this, we propose a novel post-hoc unsupervised method that automatically uncovers the concepts learned by deep models during training.
By decomposing the latent space of a layer in singular vectors and refining them by unsupervised clustering, we uncover concept vectors aligned with directions of high variance that are relevant to the model prediction, and that point to semantically distinct concepts.
Our extensive experiments reveal that the majority of our concepts are
readily understandable to humans, exhibit coherency, and bear relevance to the task at hand.
Moreover, we showcase the practical utility of our method in dataset exploration, where our concept vectors successfully identify outlier training samples affected by various confounding factors.
This novel exploration technique has remarkable versatility to data types and model architectures and it will facilitate the identification of biases and the discovery of sources of error within training data.
\end{abstract}
% \begin{keywords}
% explainability, trustworthy AI, dataset exploration, open source
% \end{keywords}
\section{Introduction}
\label{intro}
%
Understanding and explaining the inner workings of deep learning models is challenging, yet of outmost importance in high-risk applications where reliable and intuitive explanations are crucial for decision making.
Model validation plays a vital role in ensuring trustworthy predictions and to avoid actions that may negatively impact society.
Because of their ease of use and understandability, the use of user-defined queries to inquire about the relevance of high-level concepts such as objects, shapes, and textures, has shown great promise in model validation. These concept-based queries are addressed through explainability methods that leverage concept vectors, representing vectors in the latent space of a layer that are representative of a concept~\citep{kim2018interpretability}.
Methods further developing on concept vectors were proposed for multiple applications and tasks~\citep{goyal2019explaining,graziani2020concept,koh2020concept,chen2020concept}.
However, concept vectors are generally obtained as a response to user-defined queries and are, as such, biased {towards the user's knowledge and expectations, a phenomenon known as the experimenter bias~\cite{experimenter_bias}}.
Users may bias the analysis by only querying the model for some concepts while neglecting others.
The exhaustive coverage of all possible concepts, besides, is unfeasible and in some domains such as biology or chemistry, the concepts may be unclear or difficult to define.
Because of this, model interpretability with concept vectors is, as of now, limited to concepts that are well-identifiable and easy to label.
%
Achieving broader interpretability with concept vectors requires a reverse engineering approach that focuses on automating concept identification.
In this case, the only assumption made on the concepts is that they are relevant for the models' downstream task.
While a few methods have ventured in the clustering of imaging representations based on pixel similarities~\citep{ghorbani2019towards}
%\todo[inline]{Add citations here}
, only limited analyses were conducted on the factorization of layer activations~\citep{raghu2017svcca,mcgrath2022acquisition}.
These methods aim at finding a local coordinate system of the latent manifold that meaningfully explains how the network represents the training data.
{In this work,} we propose a we propose a comprehensive framework for concept discovery that can be applied across multiple architectures, tasks, and data types.
Our novel approach involves the automatic discovery of concept vectors by decomposing a layer's latent space into singular values and vectors.
By analyzing the latent space along these new axes, we identify vectors that correspond to semantically \textit{unique} and distinguishable concepts.
To evaluate the effectiveness of our approach, we apply it to the task of natural image classification. The discovered concepts align with human-understandable interpretations and demonstrate consistency with {previous} research findings.
For instance, texture and object parts emerge as prominent in vision models, reflecting the work of~\cite{kim2018interpretability} on recognizing zebra stripes and~\cite{ghorbani2019towards} on detecting vehicle parts to classify police vans.
Furthermore, user evaluation studies confirm that the {discovered concepts} are easy to understand, and show that they provide valuable insights about the model's decision making process.
%Our framework offers a generalizable and adaptable approach that empowers researchers and practitioners to develop robust and trustworthy models, thereby bridging the gap between deep learning models and human understanding.
%
\section{Related Work}
This work focuses on post-hoc interpretability, hence it aims at explaining already trained models.
Despite having acknowledged limitations~\citep{rudin2019stop,adebayo2018sanity}, post-hoc approaches allow us to gather important insights on de-facto standard models for vision tasks such as Inception~\citep{szegedy2016rethinking} and {ResNet}~\citep{he2016deep}, which are largely reused in real-world applications.
%Initially, research was mostly focused on understanding the activation patterns of individual units~\citep{bau2017network} in
Concept activation vectors (CAVs)~\citep{kim2018interpretability}, in particular, introduced the hypothesis that linear combinations of units carry information on user-defined high-level concepts.
Ghorbani et al. further extended the method to remove the need for user-defined queries~\citep{ghorbani2019towards}.
The algorithm identifies image segments that cluster in the latent space, and uses CAVs to determine if the direction of a cluster is a relevant concept for the model.
The identified vectors, however, depend on the quality of the image segments, which is not guaranteed for medical images~\citep{graziani2021sharpening}.
Moreover, CAVs are unlikely to be orthogonal in the latent space. Counterintuitively, two CAVs for unrelated concepts may be closely aligned in the latent space.
To alleviate this, \citep{chen2020concept} proposed to train a concept whitening layer that ensures the orthogonality of the concept vectors in its latent space.
On a parallel line, an analysis of deep networks based on SVD was proposed by~\citep{raghu2017svcca}, where canonical correlation is used to test the similarity of two compressed representations of different model layers.
Here the activations of a single neuron represent a vector in the canonical basis of the latent space. Following this line, non-negative matrix factorization was used to identify meaningful vectors to interpret AlphaZero~\citep{mcgrath2022acquisition}.
Our work takes inspiration from all of these works to introduce a novel method that does not rely on user-defined concepts or pixel segmentation, that does not require the training of additional concept whitening layers, and that leads to independent vectors pointing to human-understandable concepts.
\section{Methods}
\label{methods}
%
{Based on the observations of Kim et al., we assume that directions in the latent space of a layer can identify the concepts used by a model for inference~\citep{kim2018interpretability}.
However, while they rely on user-defined concept examples to find the concept vectors, here we aim to reverse-engineer the problem.
Our objective is to identify concept vectors that directly emerge as relevant directions impacting the model decision function.
We further assume, as suggested by~\citep{chen2020concept}, that concept vectors should be orthogonal to each other to maximize the separability of the concepts.}
the objective {of our method} is to identify {the} set of orthonormal vectors in the latent space that carry most of the information, i.e. that most impact {the} model's prediction.
The method consists of three phases:
\begin{enumerate}
\item {{variance alignment} {via }the singular value decomposition (SVD) of the activations of a layer};
\item {ranking of the {singular }vectors} {based on directional output sensitivity}; % high} gradients and activation values;
\item selection of the top ranked vectors {and refinement of the direction to isolate \textit{unique} concepts}. % and analysis.
\end{enumerate}
%
\subsection{Notation}\label{sec:notation}
%
We consider the prediction function of a neural network $f: \mathbb{R}^{m} \rightarrow \mathbb{R}^n$ from an $m$-dimensional input vector to an $n$-dimensional output vector.
%\uyerm{The method can be generalized to different input data types and tasks, e.g. classification, as explained in Section}~\ref{sec:cdisco-regression}
{We assume the model was {already} trained using a }dataset consist{ing} of $N$ labeled pairs of input data points and labels $\{(x_i, y_i)\}_{i=1}^N \subset \mathbb{R}^m \times \mathbb{R}^n$.
{Given an arbitrary layer $l$, the neural network can be seen as a composition of a feature extractor
$\phi^l: \mathbb{R}^m \rightarrow \mathbb{R}^d$
and a downstream predictor
$\psi^l:\mathbb{R}^d\rightarrow \mathbb{R}^n$, i.e. $f(x) = \psi^l(\phi^l(x))$.
In the following, to simplify notation, and since we consider a single layer at a time, we drop the superscript identifying the layer $l$s.}
Furthermore, for a given input $x_i$, we use the shorthand $\phi_i=\phi(x_i)$.
We are interested in finding $M$ orthonormal vectors $u_1, .., u_M \in \mathbb{R}^d$.
In the following, we present the method for $n=1$ and for densely connected layers.
The method can be extended to $n>1$, for example multi-class classification by applying the same construction to each of the outputs.
Similarly, for convolutional layers, a pooling operation is introduced to obtain a $d$-dimensional representation.
Details about both extensions are given in Section~\ref{sec:cdisco-regression}.
%
\subsection{Step 1: SVD}
%
Step one aims at identifying the vectors that best summarize the encoding of the dataset in the latent space of a layer $l$ (with dimension $d$). To this end, we apply SVD to the entire layer’s response to the input dataset.
Such matrix $\Phi \in \mathbb{R}^{d\times N}$ is obtained by column-stacking the latent representations $\phi_i$, with $i=1,\dots,N$. %=\phi^l(x_i) \in \mathbb{R}^{d}$,
%``latent data matrix''
By applying SVD, we obtain:
\begin{equation}
\Phi = U \Sigma V^T
\end{equation}
Note, $U \in \mathbb{R}^{d \times d}$, $V^T \in \mathbb{R}^{N \times N}$, and $\Sigma$ is a diagonal matrix of singular values $\sigma_1, \dots, \sigma_d$.
The left singular vectors are the columns of the matrix $U$, and they align with the variance in $\Phi$. The singular values rank the directions from the largest to the lowest observed variance.
\subsection{Step 2: {Sensitivity-based} ranking}
At this point, we are only making use of the feature extractor $\phi$, and we are not considering whether the singular vectors are actually used by the downstream predictor. This means that the vectors found in the previous step might not be particularly relevant to the downstream predictive task.
{This second step evaluates the {sensitivity} of the output function along the directions of the singular vectors. The sensitivity represents} the impact of perturbations of the feature representation of each input along the direction of the singular vector. As in~\citep{kim2018interpretability}, this is computed as the derivative of the model output along the direction of the singular vectors.
This operation is computed for all the singular vectors in $U$. Consider the gradient of the downstream predictor $\psi$ (Section~\ref{sec:notation}) with respect to the latent space, which we denote $\nabla_{\phi} \psi_i$ for input $\phi_i$.
To compute the directional derivatives, we {rotate the gradients} to align with the singular vectors in U: $\tilde{\nabla}_{\phi} \psi_{i}= U^T \nabla_{\phi} \psi_{i}$.
At this point, it is important to note that an input data point may have large gradients for low-activation features, and vice-versa, high activation values may be annihilated by close-to-zero gradients. Therefore, we consider the joint impact of gradients and activations together.
{This operation is fundamental, as it reduces the gradient noise that can be derived by small gradient values at the end of model training.}
Let
\begin{equation}
\tilde{\phi}_i = U^T\phi_i
\label{eq:activations}
\end{equation}
denote the coefficients of $\phi_i$ {rotated to align with the singular vectors}. We consider the (element-wise) product between the coefficients of the rotated activations $\tilde{\phi}_i$ and of the rotated gradients $\tilde{\nabla}_{\phi} \psi_{i}$:
\begin{equation}
\tilde{g}_i = \tilde{\nabla}_{\phi} \psi_{i} \odot \tilde{\phi}_i
\end{equation}
Finally, to compute the ranking, we consider the overall importance of a singular vector to the prediction. This is simply given by the sample mean of the ${g}_i$ over all inputs $i$ in the dataset: $\tilde{g} = \frac{1}{N}\sum_i \tilde{g}_i$. Note, $\tilde{g} \in \mathbb{R}^d$. For simplicity of the notation, we drop the tilde in the rest of the paper.
Simply put, this step replaces the conventional ranking of the singular vectors given by the values in $\Sigma$ with a new ranking based on the value of the {coefficients of the rotated gradients.}
\subsection{Extension to Vision Classification Tasks}
\label{sec:cdisco-regression}
%Our approach can be easily extended to fit a variety of data types, latent spaces and model tasks.
Let us demonstrate how to extend Steps 1 and 2 to vision classification tasks. We consider a convolutional neural network (CNN) $f: \mathbb{R}^{h'\times w' \times c'} \rightarrow \mathbb{R}^K$ classifying an input image in $n=K$ classes.
Particularly, $f_{i,k}$ is the predicted probability of input $x_i$ belonging to class $k$ , namely $ p(y=k|x=x_i)$.
$N_k$ is the number of samples in class $k$, with $N=\sum_{k=1}^K N_k$.
For convolutional layers, the feature extraction is $\phi^l: \mathbb{R}^{h'\times w' \times c} \rightarrow \mathbb{R}^{h\times w \times d}$ and it maps an input image $x_i$ to $d$ feature maps of width $w$ and height $h$.
To compute the SVD of Step 1, we aggregate the spatial information by a global average pooling operation, hence $\Phi \in \mathbb{R}^{d \times N}$, where $d$ is the number of channels. More precisely, by pooling, we reduce each $\phi(x_i) \in \mathbb{R}^{h\times w \times d}$ to a $d$-dimensional vector.
Step 2 is also modified. %to consider $K$-classification tasks with CNNs.
For each class $k$ in a $K$-classification task, we compute a separate ${g}_k \in \mathbb{R}^d$.
To obtain the ranking, we further evaluate how each ${g}_k$ compares to the values obtained for the other classes.
For instance, we compare ${g}_k$ to the distribution of the ${g}^-_k$ obtained for all the input data points that are not of class $k$, hence for all the $K$ classes except $k$.
Formally, for each class $k$ we compute the average of the projection of $g_k$ on the singular vectors:
\begin{equation}
{g}_k = \frac{1}{N_k}\sum_{i, y_i = k} U^T g_{i,k},
\end{equation}
where $g_{i,k} \in \mathbb{R}^d$ is the global average pooling of $\phi(x_i) \odot \nabla_\phi \psi_{i,k}$\footnote{Note, the pooling is here applied after the element-wise product of the features maps and the gradients}.
We then compute the sample mean and standard deviation of the values obtained for all the rest of the data, namely for all classes except $k$.
The mean is obtained, for instance, by averaging over all the samples $x_i$ such that $y_i \neq k$:
\begin{equation}
{g}^{-}_{k} = \frac{1}{N-N_{k}}\sum_{i, y_i \neq k} U^T g_{i,k'}.
\end{equation}
%Note that, after pooling, $g_k \AND g_{\bar{k}} \in \mathbb{R}^{(N-N_k) \times d}$.
Similarly, the variance is computed as:
\begin{equation}
{\sigma^{-}_{k}}^2 = \frac{1}{N-N_{k}}\sum_{i, y_i \neq k} (U^T g_{i,k'} - g^{-}_{k'})^2.
\end{equation}
Finally, the importance scores for each singular vector in $U$ are given by:
\begin{equation}
z_{k} = \frac{{g}_{k} - {g}_{k}^{-}} {\sigma^-_{k}}
\end{equation}
where $z_{k,j}$ is the importance score of the $j$-th column in $U$.
This measure is then used to obtain a class-specific ranking, from which we retain the first $M$ positions to identify $M$ concept vectors.
\subsection{Step 3: Identification of unique concept vectors}
%\todo[inline]{Laura:I think we can add here a small description of how we can isolate unique concepts by clustering. It'd be just a summary of the other paper I guess. }
%The third step identifies the first $M$ directions in the ranking as the candidate vectors for discovery.
{
This step focuses on isolating directions that act as pointers to \textit{unique} concept representations.
%Note, a unique concept refers to a distinct and individual idea or notion that is distinguishable from others.
Note, by \textit{unique concept} we refer to a distinct and individual idea or notion that is well-distinguishable from others. Therefore, we aim at isolating a specific and well-defined characteristic or pattern that is relevant to the task at hand and semantically distinguishable from other identified concepts.
At this stage, the singular vectors found by SVD provide no guarantee on whether they are pointing to unique nor human-understandable concepts.
Neural networks are known to represent more features than they have neurons~\citep{elhage2022toy}, and this causes unique features to not be disposed alongside orthogonal axes.
%meaning features may not be orthogonal.
% points towards concept representations that can be associated with a unique semantic meaning, hence exhibiting distinct characteristics or properties.
To account for this, we propose an additional step, namely the disentanglement of singular value directions into unique concept vectors that possess a distinct semantic interpretation.
This refinement process employs an unsupervised clustering methodology that was previously investigated by~\cite{o2023disentangling} on individual neuron directions.
First, we align the latent representations in accordance with the directions indicated by the singular vectors.
Next, we isolate the representations exhibiting the highest coefficients along each direction.
Hierarchical clustering is then employed to identify clusters of inputs with minimal intra-cluster distances.
The optimal number of clusters is determined based on this analysis and a distance threshold parameter is employed to control the granularity of the concepts.
To further enhance the quality of the resulting concept vectors, we employ k-means clustering with the determined number of clusters.
Additionally, we remove any outliers and insignificant clusters that contain an insufficient number of samples, for instance fewer than five images.
Candidate concept vectors are calculated as the vectors pointing to the centroids of the newly identified clusters, namely by taking the mean activations of the respective clusters.
Human-interfacing is the last action of this step, necessary to interpret the discovered knowledge.
%Visualization and interpretation of the representations pointed by the candidate vectors are important parts of .
The simplest way to obtain insights about a candidate vector is by data analysis and visualization.
Depending on the data type and the model architecture, multiple approaches can be used to analyze the data.
The simplest approach is to visualize the input data corresponding to the representations laying in the region pointed by the candidate concept vector.
In dense models, the candidate vectors can also be used as feature importance estimates.
The elements of a candidate vector $u$ are used as weights that are multiplied element-wise to the feature importance values obtained by back-propagating the model's gradients all the way to the input.
Section~\ref{sec:visualization} discusses more in detail how to obtain concept visualizations and interpretations in detail.
% For instance, one can retrieve and visualize the input data that correspond to the representations that lay in the region of the latent space that is pointed at by the candidate vector.
% project some input images, provided that they match the distribution of the data used to train the model, along the direction of the vector and retrieve input samples with increasing projection values.
%done \todo[inline]{Mara: this can be summarized to gain space}
%\subsection{Concept Visualization}
% The main challenge of concept discovery is the visualization and interpretation of the discovered information itself.
% As~\citep{mcgrath2022acquisition} discuss, the discovered information may not always be interpretable nor associable with a human-understandable concept, and research is needed in this direction.
% A concept vector discovered by our method can be interpreted as a latent discriminant direction for the downstream task.
% The variance of the activations aligns with the concept vectors, and feature perturbations along these directions have the largest impact on the output function.
\subsection{Concept Maps and Segmentation Masks}
\label{sec:visualization}
The main challenge of concept discovery is the visualization and interpretation of the discovered information itself.
As~\citep{mcgrath2022acquisition} discuss, the discovered information may not always be interpretable nor associable with a human-understandable concept, and research is needed in this direction.
A concept vector discovered by our method can be interpreted as a latent discriminant direction for the downstream task.
The variance of the activations aligns with the concept vectors, and feature perturbations along these directions have the largest impact on the output function.
Visualizing how a concept activates given a certain input can be particularly beneficial to the interpretation of the discovered information in human-understandable terms.
We define concept activation maps as the visualization of the model's response, for a given concept, to a certain input.
The computation of such maps is straightforward for convolutional networks.
For a discovered concept direction $u\in \mathbb{R}^{d}$, and an input image $x_i$, we can visualize the layer's activation responding to the discovered concept as the sum of the $d$ feature maps for each channel $\phi_{i,1},\dots, \phi_{i,d}$ weighed by the concept vector coefficients.
% We compute a projection of the feature maps on the concept vector direction:
% \begin{equation}
% \sum_{j=0}^d u_j \phi_{i,j}
% \end{equation}
In other words, one can take the feature maps of each channel and weight them by the coefficient values of the concept vector\footnote{Being the concept vector a singular vector, it has norm of one.}.
Note, if we were to consider a concept that is fully aligned with the $n$-th feature map (with $n<d$, e.g.. $u = (0,...,0,1,0,...,0)$ (non-zero only for $u_n$), the visualization for $x_i$ would correspond to the $n$-th feature map itself.
Concept segmentation masks can be directly derived by the concept maps as in~\citep{bau2017network}, namely by retaining the input pixel values with concept activation higher than the 80-th percentile in the concept maps.
%
% \begin{figure*}[!t]
% \centering
% \includegraphics[width=\linewidth]{figures/conceptviswmax.png}
% \caption{Visualization of the discovered concept vectors for ImageNet classes. In the first two rows, the input image is shown together with a zoomed-in version of the automatically segmented concept. The last row shows the input images with largest projection on the concept vectors and the relative concept segmentation masks. Our results are in line with previously discovered concepts for these classes.}
% \label{fig:conceptvis-results-general}
% \end{figure*}
%
\subsection{Dataset Exploration and Outlier Detection}{
We showcase the utility of concept discovery for dataset exploration, demonstrating how the discovered concept vectors can be used to find input samples that are mislabeled or that contain confounding factors.
%For instance, we identify % the representation of the input samples in the latent space spanned by the concept vectors.
The data is rotated to align in the space spanned by the singular vectors and anomalous samples are identified based on the statistical dispersion of the data
For instance, we isolate 10\% of the representations that fall outside of the interquartile range of the representation coefficients for the training data.
The inputs corresponding to these representations are flagged for further inspection, as they are outliers that may contain artefacts or potentially misleading confounding factors. }
\section{Results}
%
This section presents the comprehensive evaluation of our concept discovery approach across various tasks and models. Our experiments focus on widely adopted models with publicly available pre-trained weights, which have been extensively studied in the field of interpretability.
Specifically, we present the results obtained using Inception V3 (IV3)~\citep{szegedy2016rethinking} with pretrained weights on the ImageNet ILSVRC2012 dataset (Russakovsky et al., 2015)~\citep{russakovsky2015imagenet}.
In addition, we demonstrate the versatility of concept discovery beyond image-based applications by showcasing its effectiveness in classification tasks involving tabular data (details provided in the Supplementary Materials).
% NO NEED
% We begin by providing empirical evidence that our method automatically identifies concepts associated with natural image categories in Sections~\ref{sec:discoveredconcepts} and~\ref{sec:granular-concepts}. Subsequently, we delve into an analysis of whether the representations indicated by singular vectors correspond to unique concepts. We address the challenge of disentangling directions with multiple semantics and transform them into distinct concept vectors in~Section~\ref{sec:disentanglement}.
% To evaluate the interpretability and clarity of the discovered concepts, we conduct qualitative assessments through human evaluation studies in Section~\ref{sec:humaneval}. Furthermore, we present quantitative evaluations by measuring the performance drop when removing the discovered concepts from the representations in Section~\ref{sec:cremoval}, thereby highlighting the significance of these concepts.
% Lastly, we showcase the practical utility of our method for automatic outlier detection and dataset exploration in Section~\ref{sec:exploration}, illustrating its effectiveness in identifying anomalies within the input data.
%%
\begin{figure}
\centering
\includegraphics[width=\linewidth]{journalfigs/cdisco.results.1.jpg}
\caption{A. 2D UMAP of images activating the discovered concept vectors for \textit{bubble}, \textit{police van} and \textit{zebra}. The images form three seperate clusters. We visualize concept segmentation masks for the images in each of the clusters. B. Concept segmentation masks and maximally activating input image for the class \textit{bubble}. C. Concept segmentation masks for \textit{zebra}. The concepts discovered in B. and C. are \textit{unique} concepts. D. A bisemantic concept is identified for the class \textit{police van}, activating for both vehicle parts and dog paws. After the refinement through clustering, the concept is split in two separate vectors representing either the graphics of the police logo or the dog fur. E. Unique concept representing the police log on the van structure. F. Unique concept deriving from the disentanglement of the polysemantic concept in D. }
\label{fig:conceptvis-results-general}
\end{figure}
\subsection{Confirmation of existing findings in discovered concepts}
%confirm existing findings}
\label{sec:discoveredconcepts}
%
Our method automatically identifies concepts associated with image categories. Our results are in line with specific findings that were already discussed by previous research~\citep{kim2018interpretability,ghorbani2019towards}, specifically for the object categories of \textit{zebra, police van and bubble}.
Additional results for other classes than the selected ones and for ResNet50 are provided in the Supplementary Materials~\ref{apn:ResNet50} and an integral analysis of the concepts discovered for ImageNet classes will be made accessible online.
We identify three orthogonal vectors as candidate concept vectors (i.e. one for each of the analyzed classes, with $M=1$).
The discovered concepts encompass a diverse range of visual elements, including patterns observed in the coat of \textit{zebra}, graphical features and tires associated with a \textit{police van}, and glossy reflections as seen in a \textit{bubble}, as demonstrated in Figure~\ref{fig:conceptvis-results-general}.
The maximally activating images are either visually similar or exaggerate the motifs shown by the activation maps.
The maximally activating input for \textit{bubble}, for example, exaggerates the glossy-like reflections and the round shapes generally observed in soap bubbles.
The text of the menu restaurant contains black and white stripes, which are also in \textit{zebra} images.
Importantly, these illustrations were obtained without any prior annotation of the concepts as detailed in Section~\ref{sec:visualization}.
Upon closer examination of the \textit{police van} class (Figure\ref{fig:conceptvis-results-general}D, our analysis reveals an interesting observation. The candidate concept vector identified for this class exhibits responses to multiple unrelated features. In addition to activating for van tires and police graphics, it also responds to dog chests, which may seem unexpected at first. However, this phenomenon is not entirely unexpected and it can be attributed to the concept of \emph{superposition}, which has been observed in CNNs~\citep{circuits}.
Superposition refers to the ability of neural networks to represent more features than the number of neurons they possess. This results in the presence of \textit{polysemantic neurons}, which exhibit activation patterns that are semantically unrelated. Previous research highlighted the existence of such \textit{polysemantic} neurons that activate simultaneously for car hoods and animal paws~\citep{circuits,o2023disentangling}, aligning with our findings.
%
Our method demonstrates its capability to successfully disentangle the two activations associated with the \textit{police van} class by identifying two distinct clusters that refine the concept direction into separate vectors, as illustrated in Figures~\ref{fig:conceptvis-results-general}E and F. In this process, the orthogonality of the direction is compromised to achieve unique concept directions for each activation. The disentanglement of the two patters allows us to capture the distinct semantic meanings associated with each activation, providing a more refined and comprehensive understanding of the underlying concepts.
%MOVE LATER ON Importantly, the polysemantic nature of neurons is inherent in their activation patterns, and this characteristic extends to all the concept discovery methods that use matrix factorization to identify
%the orthogonal singular value decomposition (SVD) directions we employ in our analysis.
% MOVE TO DISCUSSION Our approach enables automatic segmentation at various scales, and the segmentation process remains robust even in cases where a concept appears multiple times within a single image.
\subsection{Coherence and human interpretability of discovered concepts}
%concepts are coherent and human-interpretable}
\label{sec:humaneval}
%
To assess the coherence and human interpretability of the discovered concepts, we conducted a human evaluation test following the methodology outlined in previous works~\citep{ghorbani2019towards}.
The test involved 30 participants, and the details of the evaluation test can be accessed at the following link: \url{https://forms.gle/MJ63G984ERvozuF38}.
%
The evaluation comprised three main experiments, namely (i) intruder detection (ii) concept meaningfulness and (iii) inter-user agreement.
Participants were familiarized with the test format by a prior introduction with an exemplar question and the corresponding correct answer.
In the intruder detection experiment (i) illustrated in Figure~\ref{fig:results-p1}A, participants were presented with ten questions, each featuring a visualization of four concept segmentation masks associated with a specific concept vector, and one random mask from a different concept vector. The task was to identify the outlier image that conceptually differed from the others.
Participants successfully recognized the intruders with average accuracy of $0.88$. Concept meaningfulness (ii) was measured by asking participants to label the concepts based on the visualization of the concept segmentation masks for three images.
Despite the small image segments depicted the concept segmentation masks, all participants accurately labeled all the images, indicating that the concept segmentation masks were easily interpretable and could be associated with specific concepts.
Inter-user agreement (iii) was evaluated by requesting participants to agree or disagree with concept labels provided by other participants. Out of the 30 users, inter-user agreement exceeded 91\% for all questions. Notably, concepts identified for dog breeds showed relatively lower agreement scores, as users disagreed on the specific type of animal depicted in the concept segmentation masks.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{journalfigs/cdisco.results.2.jpg}
\caption{A. Detailed results of the intruder detection experiment with human participants. Out of ten classes, participants were able to detect with very high accuracy the intruding concept. B. Wordclouds of the concept labelling results for two concepts. The font size corresponds to the word frequency. C. Concept importance measured in terms of the impact of removing concepts in the input space by occlusion, namely by masking out areas with high values in the concept activation masks. Performance is evaluated on all ImageNet classes while gradually removing concepts starting from the most important. D. Concept importance measured as the impact of removing concepts in the latent space by annihilating the model weights that are linked to the concept. Performance is evaluated on all ImageNet classes while gradually removing concepts starting from the most important. E. Fine-grained concept activation maps for the first, second and third concept vectors differentiating rugby from soccer balls. The map adopts a symmetric colorbar with red for positive-, blue for negative- and green for zero-valued activations. Images are sorted from the largest absolute projection values on the concept vector.
F. Fine-grained concept activation maps with symmetric colorbar for the concept activation maps on ImageWoof categories.}
\label{fig:results-p1}
\end{figure}
\subsection{Quantitative evaluation of concept relevance}% through quantitative evaluation} %confirms concept relevance}
\label{sec:cremoval}
We measure the significance of the discovered concept vectors by quantifying the impact of concept removal using two different approaches, namely occlusion in the input space and weight annihilation at the intermediate layer.
Following the methodology of~\citep{ghorbani2019towards}, occlusion is applied to the input pixels with high values in the concept map of each concept, specifically targeting the pixels with values above the 80th percentile.
We quantify the smallest destroying concepts (SDC) as the smallest number of concepts to remove in order to see a performance degradation on at least $80$\% of the dataset. Starting with the most important concept, we gradually remove concepts while monitoring the drop in performance.
As depicted in Figure~\ref{fig:results-p1}C, we observe a degradation of at least $80$\% in the prediction accuracy when the first concept is removed for nearly $400$ classes.
The performance degradation extends to almost all ImageNet classes ($962$ out of $1000$) when we remove the first five concepts.
Supplementary Figure~\ref{apnfig:sdcimagewoof} in the Supplementary Materials provides a comparison the model predictions before and after the removal of the SDC for each class.
One limitation of SDC is is the introduction of a distribution shift in the modified images due to the replacement of pixels in the input space. It becomes challenging to disentangle whether the prediction change is solely due to removing the concept or the distribution shift~\citep{hooker2019benchmark}.
To address this concern, we conduct an additional analysis where concepts are dropped directly inside the layer, comparing the change in prediction accuracy against dropping random directions.
To drop a concept, we retain the layer weights with low concept vector components and we set the remaining weights to zero.
For example, to remove the first concept of the class \textit{lionfish}, we set to zero $204$ parameters out of the original $2,048$ (keeping the remaining $1,844$ weight values untouched), based on the $80$-th percentile of the components of the concept vector for that class.
We compare the impact of this ablation against dropping random weights to disentangle the effect of removing a specific concept direction from randomly setting weights to zero.
Figure~\ref{fig:results-p1}D shows the accuracy drop observed when removing the concept directions from the first most important to the fifth, computed on the training images and 1000 validation images.
Removing the first five concept directions causes an accuracy drop from $0.920$ to $0.785$ on train data, and from $0.762$ to $0.676$ on test data.
The same drop cannot be observed instead when setting random weights to zero (i.e. accuracy $0.89$ on train and $0.734$ on test data).
%
\subsection{Fine-grained analyzes}
\label{sec:granular-concepts}
Concept discovery can be conducted in a fine-grained manner, focusing on specific subsets of classes. This targeted approach enables the identification of concepts that are tailored and relevant to a smaller group of similar classes. Such granular analyses are particularly valuable for classes that pose classification challenges or hold specific importance for interpretability analysis.
We consider a subset of ten dog breeds classes in ImageNet, referred as ImageWoof~\citep{imagenette}. This subset presents a significant challenge as the differences between the dog breeds are subtle (more details in the Supplementary Materials~\ref{apn:imagewoof}).
Our method identifies one concept for each class, totalling ten concepts.
For \textit{Dingo, Beagle and Old English sheepdog} we visualize the images that maximally activate the concept for each class, hence with the largest projection on the concept vector (Figure~\ref{fig:results-p1}G).
High activation values are around the head for all dogs, and confusing factors are ignored (e.g. the woman in \textit{sheepdog}).
Each dog has a very distinctive head shape (the \textit{dingo} has straight ears, the beagle long bent ears and the \textit{sheepdog} has long black and white fur) and for this reason, different concept vectors are found for each class.
The \textit{beagle} also shows a diffused attention that includes the tail and paws of the dogs. This is probably to distinguish this class from the \textit{English foxhound}.
By running concept discovery on pairs of classes, we identify concepts that provide informative insights into what distinguishes one class from another. For instance, Figures~\ref{fig:results-p1}E and F showcase the concepts that differentiate a \textit{Soccer ball} from a \textit{Rugby ball}. In this analysis, we set $M=3$ and present three discovered concepts for each class in order of importance.
The most important concept for \textit{Rugby ball} emerges to be the shape of the ball, as shown by the activations around the ball in Figure~\ref{fig:results-p1}E.
On the other hand, the concept activation maps for \textit{Soccer ball} identify the knitted pattern of the ball as the most important concept, and the soccer ball shape emerges only as the second most important concept.
We can also see the attention to the context, particularly in the third concept. The maps activate on the players on the field, showing sensitivity to the player's jerseys and socks.
Some of these findings align with the hypotheses found in~\citep{ghorbani2019towards} about the relevance of jerseys to classify images in the \textit{basketball} category.
Thus, jerseys and socks may introduce confounding factors in the classification of these categories.
Finally, in the inputs for the \textit{rugby} class, the maps also activate on the players' hands holding the ball, or on the act of tackling, whereas in the \textit{soccer} images the maps activate on the acts of kicking the ball.
%\subsection{Model insights}
\subsection{Concept Uniqueness}
\label{sec:disentanglement}
%
{We evaluate the uniqueness of the concepts identified by our method, showing that it is possible to disentangle superposition and identify vectors that point to unique concepts.}
% The first thing to note is that the discovered concepts do not align with individual model units.
% The alignment can be computed as
% The cosine similarity between the discovered concept vectors for each category in ImageNet (we consider, for simplicity $M=1$ and 100 concept vectors in total) and the basis vector of the latent space is, in fact, always lower than 0.5 for the layers \textit{Mixed\_7b}, as well as for \textit{Mixed\_5b}, \textit{Mixed\_5d}, \textit{Mixed\_6c}, and \textit{Mixed\_7c}.
% Note, a cosine similarity close to zero suggests the independence of the two directions, whereas had the vectors been aligned, the absolute value would have been close to one.
% If the cosine similarity is close to one in absolute value, then the two directions align perfectly.
% A cosine similarity of .
Firstly, it is important to note that the discovered concepts do not align with individual model units. To quantify the alignment, we compute the cosine similarity between the discovered concept vectors for each category in ImageNet (considering $M=1$ and a total of 100 concept vectors) and the basis vector of the latent space.
The cosine similarity is consistently lower than 0.5 for various layers such as \textit{Mixed\_7b}, \textit{Mixed\_5b}, \textit{Mixed\_5d}, \textit{Mixed\_6c}, and \textit{Mixed\_7c}. A cosine similarity close to zero indicates the independence of the two directions, suggesting that the concept vectors and the latent space vectors are not aligned. In contrast, if the vectors were aligned, the absolute value of the cosine similarity would be close to one.
Next, we examine the outcomes of applying the disentanglement method proposed in \cite{o2023disentangling}.
Specifically, we compare how many distinct clusters are suggested by the disentanglement method when applied to individual neurons as opposed to the singular vectors identified by singular value decomposition in our method.
This analysis is informative on whether the basis of the singular vectors can facilitate capturing inherently distinct concept representations, i.e. \textit{unique} concepts.
Figure \ref{fig:results-p2}A, for example, illustrates how three separate clusters are identified as unique concepts starting from one of the singular vectors.
The disentanglement outcome provides us with 3 distinct concepts for babies, teddy bears, and necklaces.
By applying this approach to all of the singular vector directions within layer \textit{Mixed\_7b}, we quantify how many singular vector directions were already pointing to unique clusters and how many required disentanglement.
As shown by Figures~\ref{fig:results-p2}B and C, most of the singular vectors in our method already point to unique concept directions, and only a few require disentanglement. As opposed to individual neuron directions, fewer directions show polysemanticity.
This means that the singular value decomposition step in our method is a necessary approach that facilitates the identification of unique concepts.
The remaining singular vector directions that are not unique are then disentangled by our clustering method in distinct concepts.
\begin{figure}
\centering
\includegraphics[width=\textwidth]{journalfigs/cdisco.results.3.jpg}
\caption{A. UMAP of the images maximally activating a polysemantic singular vector. Hierarchical clustering is used to identify the optimal number of clusters, i.e. three. Three concept vectors pointing to the centroids of each cluster are thus derived to obtain unique pointers to concepts. B. Number of clusters identified by hierarchical clustering for the images maximally activating individual neurons. The high number of neurons with at least two clusters shows that the directions are highly polysemantic, confirming existing research. C. Number of clusters identified by hierarchical clustering for the images that maximally activate our singular vectors. The high number of directions with a single cluster shows that the singular vectors are more likely to identify unisemantic concepts. }
\label{fig:results-p2}
\end{figure}
% Applying we observed the following outcome:
% 488 directions were already originally associated with a single concept, 157 directions encompassed two concepts, 99 directions yielded three concepts, 30 directions revealed four distinct concepts, 11 directions contained five concepts, and 3 directions manifested six distinct concepts.
\subsection{Effective Outlier Detection}
\label{sec:exploration}
{
Our method successfully identifies $23$\% (equivalent $0.2$\% of the ImageNet training set) of the training images as outliers based on their representation in the latent space. Among these images, $184$ ($1.8$\% of the training set) are misclassified by the model.
Note, the flagged images were seen during training, hence they are challenging training examples that confuse the model rather than improving its robustness. This observation is supported by the lower top-1 accuracy on the flagged images at $0.90$ compared to $0.93$ on the rest of the training images.
Furthermore, when compared to random neurons or random directions, our method demonstrates higher accuracy in identifying these challenging training examples.
The accuracy on the flagged images obtained through our method is $0.90$, whereas random neurons or vectors achieve an accuracy of $0.92$. This reaffirms the notion that the concept vectors discovered by our method offer more informative directions for dataset exploration than random alternatives.
Figure~\ref{fig:trainingpath} shows some examples of the outliers identified by our method. These images present notable variations in style and resolution (e.g., the first two images from the left) or contain confounding factors. For example, in the third image from the left, multiple labels (shovel and lawn mower) are equally correct. Additional examples are shown in the Supplementary Materials. }
\begin{figure}
\centering
\includegraphics[width=.8\linewidth]{journalfigs/cdisco.results.4.jpg}
\caption{A. Results of dataset exploration. Our method identifies training images with distinct issues, such as style shift to drawings (1st image), poor resolution (2nd), multiple correct labels (3rd) and optical illusions (4th).}
%The method identifies training images with particular issues. The first example presents a strong style shift from real images to drawings. Extremely poor resolution affects the quality of the second input. The last two images present confounding factors. Multiple labels are equally correct for the third image, and the last image shows an optical illusion. Where there seems to be a cliff, there is actually a high resolution detail of two ants on a wooden surface.}
\label{fig:trainingpath}
\end{figure}
\section{Discussion}
The automatic discovery of the latent concepts used by complex models is a challenging research direction that is still under-explored.
We presented a novel approach that is simple yet effective to automatically discover concept vectors that point at image clusters with a unique semantic meaning.
A key strength in our method is that it does not require manual annotation or supervision. The automated discovery process saves significant time and effort compared to traditional manual labeling methods.
Besides, we demonstrate that the discovered concepts are relevant for model performance by evaluating the impact of concept removal. As expected, removing the concepts shows a significant degradation of the model's prediction accuracy.
The uniqueness of the discovered concepts is another noteworthy aspect of our approach. We demonstrate that our concept vectors can more appropriately capture independent and distinct visual attributes than individual neuron directions.
It is important to note that our approach is not without limitations. The reliance on the input data used to identify the concepts is a main limitation. If the network fails to capture relevant concepts during training, our method may not be able to discover them effectively. Additionally, the evaluation of concept meaningfulness relies on human participants, which finally introduces some degree of subjectivity.
Notwithstanding, our method proves to be useful in detecting input images with confounding issues, showing that relevant insights can be gathered by looking at the concept directions. The flagged outliers highlight images that exhibit variations in style, resolution, or contain confounding factors. This information is valuable for improving model robustness and understanding the limitations and challenges associated with specific classes or image characteristics.
Ultimately, we believe that this work can impact the identification of patterns in fields where deep learning models are accelerating knowledge discovery such as chemistry and biology.
%\bibliography{main}
\bibliography{main}
\bibliographystyle{tmlr}
%\newpage
\appendix
%\appendixpage
\subfile{appendix.tex}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\renewcommand\thefigure{\thesection.\arabic{figure}}
\section{Additional Results on IV3 and ResNet50}
\label{apn:ResNet50}
Note: In our study, we apply the methodology to the 1000 classes in ImageNet. To ensure computational feasibility and accommodate our infrastructure capabilities, we employ an undersampling technique. Specifically, we retain only one out of every ten images from the ImageNet training dataset. This strategic choice enables us to manage memory requirements effectively and streamline the computational process.
%To limit the memory requirements and make the computation accessible to our infrastructure, we undersample the ImageNet training dataset by retaining only one of every ten images.
%T%his choice keeps the computation feasible and light.
Nevertheless, we demonstrate how concepts can also be obtained by utilizing the complete set of available images for select classes, providing a more granular analysis in Section~\ref{sec:granular-concepts}.
Throughout the experiments, unless explicitly stated otherwise, we primarily focus on the concatenation layer named \textit{Mixed 7b}. This layer is situated deep within the model, close to the end, and is expected to capture complex and high-level concepts. However, we emphasize that similar analyses can be conducted at different depths, layers, and even on entirely different architectures, as showcased in Appendix~\ref{apn:ResNet50}.
We provide additional results for 16 classes, namely \textit{airliner, clock, corkscrew, albatross, border collie, road sign, flamingo, mushroom,artichoke, hammerhead shark, screwdriver, iPod, tench fish, suspension bridge, umbrella} and {cucumber}. The concept segmentation masks for the first most important concept are shown in Figure~\ref{apnfig:addvis}. The concepts were resized to fit in the square, but they originally appear at multiple scales in the input images.
Results obtained on ResNet50 (at \textit{layer\_4.0.add}) are shown in Figures~\ref{apnfig:resnet50vis} and~\ref{apnfig:maxresnet}.
The visualizations point to concepts that are similar to those highlighted in IV3 visualizations, such as the fish fins, the zebra stripes, the glass-like reflections and van tires and logos.
Figure~\ref{apnfig:maxresnet} shows the input images in the dataset that have the largest projection value on the concept vector for the four analysed classes.
We can see the striped pattern emerging again for the class zebra, although in this case the color of the stripes seems to be given less importance. For the class \textit{bubble}, the shape of the bird belly and their repeated presence may share a similarity with the images in the bubble class.
\begin{figure}[!h]
\centering
\includegraphics[width=\linewidth]{apnfigs/maxcresnet.png}
\caption{Results on ResNet50. Visualization of the inputs with the largest projection value on the concept vectors for the classes \textit{lionfish, police van, bubble and zebra}. Next to the input images we visualize the concept segmentation masks.}
\label{apnfig:maxresnet}
\end{figure}
%%%%%%%%%
\section{Additional Results on ImageWoof}
\label{apn:imagewoof}
The ImageWoof dataset is a subset of the following ImageNet classes: \textit{Australian terrier, Border terrier, Samoyed, Beagle, Shih-Tzu, English foxhound, Rhodesian ridgeback, Dingo, Golden retriever, Old English sheepdog}.
An input image for each class is shown in Figure~\ref{apnfig:dogs}.
This dataset is challenging to classify because the classes represent multiple dog breeds with subtle differences.
As Figure~\ref{apnfig:confmatimagewoof} shows, the confusion matrix obtained on \textit{training} images also presents misclassified samples.
%The dataset can be downloaded at \url{https://github.com/fastai/imagenette}.
\begin{figure}[!h]
\centering
\includegraphics[width=0.8\linewidth]{apnfigs/confmatimagewoof.png}
\caption{Detail of the IV3 confusion matrix on ImageWoof classes. }
\label{apnfig:confmatimagewoof}
\end{figure}
Figure~\ref{apnfig:imagewoofvis} shows additional concept maps for the dog breeds ($M=1$). %We show in Figure~\ref{apnfig:imagewoofmax} the inputs maximally activating the concept for each class.
\begin{figure}
\centering
\includegraphics[width=0.5\linewidth]{apnfigs/dogs1.png}
\includegraphics[width=0.5\linewidth]{apnfigs/dogs2.png}
\caption{Concept maps for the classes in ImageWoof. The images are ranked in order of their projection value on the concept vector (largest to lowest). }
\label{apnfig:imagewoofvis}
\end{figure}
We show the prediction change before and after the removal of the smallest number of destroying concepts on ImageWoof in Figure~\ref{apnfig:sdcimagewoof}.
No prediction change would appear as a diagonal line, whereas in our case we can clearly see that the prediction becomes random after removing the concept.
The SDC is 1 for \textit{Shih-Tzu, Rhodesian ridgeback, Beagle, Australian terrier and Golden retriever}, 2 for the \textit{Old English sheepdog}, 3 for \textit{Samoyed} and \textit{Dingo} and 4 for the \textit{Border Terrier}.
% \begin{figure}[b]
% \centering
% \includegraphics[width=\linewidth]{figures/labelinput.png}
% \caption{ImageWoof labels against IV3 predictions after concept removal in the input space.}
% \label{apnfig:sdcimagewooflabels}
% \end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{apnfigs/sdcpredictioninput.png}
\caption{IV3 predictions on ImageWoof before (original) and after concept removal in the input space.}
\label{apnfig:sdcimagewoof}
\end{figure}
% \begin{figure}[b]
% \centering
% \includegraphics[width=\linewidth]{figures/sdcpredact.png}
% \caption{(to recompute/wrong axes) IV3 predictions on ImageWoof before (original) and after concept removal in the latent space.}
% \label{apnfig:predchangelatentimagewof}
% \end{figure}
%\subsection{ResNet50 concepts}
\section{Additional Results from the User Evaluation}
% Here the detailed results of the user evaluation study.
Figure~\ref{apnfig:intruderextra} reports in detail the performance of the end users on each question in the evaluation study part (i).
\begin{figure}
\centering
\includegraphics[width=\linewidth]{apnfigs/intruderextra.png}
\caption{Detailed performance on part i. of the human evaluation study.}
\label{apnfig:intruderextra}
\end{figure}
%\section{Superposition}
\section{Benchmark results on COMPAS}
% \subsection{Concept Discovery for tabular data}
% \label{sec:compas}
Here we demonstrate the applicability of concept discovery to non-imaging applications such as COMPAS. The model is, in this case, a densely connected feed-forward network.
The COMPAS system uses criminal records and demographic features of nearly 19 thousand defendants to predict the likelihood of a new offense.
Previous studies largely discussed whether race, gender and age of the defendants are sensible variables affecting the system's fairness~\cite{compas,rudin2018age}.
While we agree that a multilayer perceptron is unnecessarily complex for this task~\cite{rudin2019stop}, we recognize its importance as a contradictory task in the model interpretability literature.
Figure~\ref{fig:compas} shows three discovered concepts for this task ($M=3$). Note, gradient backpropagation is used to visualize the importance of each input feature for the concept vector.
The values in the plot represent the importance given to the input features to project any input on the concept vector.
The three concepts are a linear combination of multiple input features, and high relevance is given to the age of the defendant (age, in the first plot), the number of prior arrests (no\_priors, in all plots), and the length of previous detentions (length\_stay, two plots on the right).
The concept scores in the plot give a global explanation of the model. Local explanations about single inputs can be obtained by multiplying the scores with the feature values of each input.
We benchmark the faithfulness of our explanations against other explainability methods by the Prediction Gap on Important feature perturbation (PGI) and Unimportant feature perturbation (PGU), as in~\cite{agarwal2022openxai}.
High PGI and low PGU values are desirable. The former indicates that the explanation identifies important features, while the latter that the method ignores unimportant ones.
We compute the PGI and PGU following the benchmark code~\footnote{\url{github.com/AI4LIFE-GROUP/OpenXAI/blob/main/OpenXAI\%20quickstart.ipynb}}, obtaining $0.27 $ PGI and $0.0026$ PGU.
These values are comparable with other standard explainability methods, outperforming the benchmark results obtained by gradient-based approaches and LIME, as shown in the Appendix Table~\ref{apntab:compas}.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{apnfigs/compasdef.png}
\caption{Top 3 concepts discovered on COMPAS. The bar values illustrate the feature importance values to project each input on the discovered concept vector.}
\label{fig:compas}
\end{figure}
%
Table~\ref{apntab:compas} benchmarks our method in the OpenXAI benchmark~\cite{agarwal2022openxai}. The results in the table are taken, except for concept discovery, from the online leaderboard of the benchmark at ~\url{open-xai.github.io}.
The hyperparameters of the explainer were set as illustrated in the benchmark instructions, i.e. \{\texttt{protected\_class:1}; \texttt{positive\_outcome:1}; \texttt{perturbation\_std:0.3}\}.
\begin{table}[!h]
\centering
\caption{Explanation faithfulness for COMPAS.}
\label{apntab:compas}
\begin{tabular}{c|c c}
Method & PGI $\uparrow$ & PGU $\downarrow$ \\\hline
LIME & 0.232 & 0.247 \\
Vanilla Gradient & 0.240& 0.240\\
Integrated Gradient &0.240& 0.238 \\
Gradient x Input & 0.254 & 0.216\\
SHAP & {0.274} & 0.194 \\
SmoothGrad & \textbf{0.324} & 0.106 \\\hline
\textbf{concept discovery} & 0.268 & \textbf{0.0026} \\
\end{tabular}
\end{table}
\section{Additional Visualizations}
\begin{figure*}[!t]
\centering
\includegraphics[width=\linewidth]{apnfigs/imagewoof_classes.png}
\caption{Dog breeds in ImageWoof. }
\label{apnfig:dogs}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{apnfigs/addvis1.jpg}
\includegraphics[width=\linewidth]{apnfigs/addvis2.jpg}
\caption{IV3 concepts for additional classes. We show the resized segmentation masks.}
\label{apnfig:addvis}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=\linewidth]{apnfigs/resnet50vis.png}
\caption{Results on RestNet50. Segmentation masks of the concept vectors in \textit{layer4.add} of ResNet50 for the same classes and input images in Figure~\ref{fig:conceptvis-results-general}.}
\label{apnfig:resnet50vis}
\end{figure*}
\begin{thebibliography}{24}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Adebayo et~al.(2018)Adebayo, Gilmer, Muelly, Goodfellow, Hardt, and
Kim]{adebayo2018sanity}
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt,
and Been Kim.
\newblock Sanity checks for saliency maps.
\newblock \emph{Advances in neural information processing systems}, 31, 2018.
\bibitem[Agarwal et~al.(2022)Agarwal, Krishna, Saxena, Pawelczyk, Johnson,
Puri, Zitnik, and Lakkaraju]{agarwal2022openxai}
Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari
Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju.
\newblock Open{XAI}: Towards a transparent evaluation of model explanations.
\newblock In \emph{Thirty-sixth Conference on Neural Information Processing
Systems Datasets and Benchmarks Track}, 2022.
\newblock URL \url{https://openreview.net/forum?id=MU2495w47rz}.
\bibitem[Bau et~al.(2017)Bau, Zhou, Khosla, Oliva, and
Torralba]{bau2017network}
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.
\newblock Network dissection: Quantifying interpretability of deep visual
representations.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 6541--6549, 2017.
\bibitem[Chen et~al.(2020)Chen, Bei, and Rudin]{chen2020concept}
Zhi Chen, Yijie Bei, and Cynthia Rudin.
\newblock Concept whitening for interpretable image recognition.
\newblock \emph{Nature Machine Intelligence}, 2\penalty0 (12):\penalty0
772--782, 2020.
\bibitem[Elhage et~al.(2022)Elhage, Hume, Olsson, Schiefer, Henighan, Kravec,
Hatfield-Dodds, Lasenby, Drain, Chen, et~al.]{elhage2022toy}
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan,
Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen,
et~al.
\newblock Toy models of superposition.
\newblock \emph{arXiv preprint arXiv:2209.10652}, 2022.
\bibitem[Ghorbani et~al.(2019)Ghorbani, Wexler, Zou, and
Kim]{ghorbani2019towards}
Amirata Ghorbani, James Wexler, James~Y Zou, and Been Kim.
\newblock Towards automatic concept-based explanations.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Goyal et~al.(2019)Goyal, Feder, Shalit, and Kim]{goyal2019explaining}
Yash Goyal, Amir Feder, Uri Shalit, and Been Kim.
\newblock Explaining classifiers with causal concept effect (cace).
\newblock \emph{arXiv preprint arXiv:1907.07165}, 2019.
\bibitem[Graziani et~al.(2020)Graziani, Andrearczyk, Marchand-Maillet, and
Müller]{graziani2020concept}
Mara Graziani, Vincent Andrearczyk, Stephane Marchand-Maillet, and Henning
Müller.
\newblock Concept attribution: Explaining cnn decisions to physicians.
\newblock \emph{Computers in Biology and Medicine}, 123:\penalty0 103865, 2020.
\newblock ISSN 0010-4825.
\newblock \doi{https://doi.org/10.1016/j.compbiomed.2020.103865}.
\bibitem[Graziani et~al.(2021)Graziani, Palatnik~de Sousa, Vellasco, Costa~da
Silva, M{\"u}ller, and Andrearczyk]{graziani2021sharpening}
Mara Graziani, Iam Palatnik~de Sousa, Marley~MBR Vellasco, Eduardo Costa~da
Silva, Henning M{\"u}ller, and Vincent Andrearczyk.
\newblock Sharpening local interpretable model-agnostic explanations for
histopathology: Improved understandability and reliability.
\newblock In \emph{International Conference on Medical Image Computing and
Computer-Assisted Intervention}, pp.\ 540--549. Springer, 2021.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 770--778, 2016.
\bibitem[Hooker et~al.(2019)Hooker, Erhan, Kindermans, and
Kim]{hooker2019benchmark}
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim.
\newblock A benchmark for interpretability methods in deep neural networks.
\newblock \emph{Advances in neural information processing systems}, 32, 2019.
\bibitem[Howard()]{imagenette}
Jeremy Howard.
\newblock imagenette.
\newblock URL \url{https://github.com/fastai/imagenette/}.
\bibitem[Jordan \& Freiburger(2015)Jordan and Freiburger]{compas}
Kareem~L Jordan and Tina~L Freiburger.
\newblock The effect of race/ethnicity on sentencing: Examining sentence type,
jail length, and prison length.
\newblock \emph{Journal of Ethnicity in Criminal Justice}, 13\penalty0
(3):\penalty0 179--196, 2015.
\bibitem[Kim et~al.(2018)Kim, Wattenberg, Gilmer, Cai, Wexler, Viegas,
et~al.]{kim2018interpretability}
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda
Viegas, et~al.
\newblock Interpretability beyond feature attribution: Quantitative testing
with concept activation vectors (tcav).
\newblock In \emph{International conference on machine learning}, pp.\
2668--2677. PMLR, 2018.
\bibitem[Koh et~al.(2020)Koh, Nguyen, Tang, Mussmann, Pierson, Kim, and
Liang]{koh2020concept}
Pang~Wei Koh, Thao Nguyen, Yew~Siang Tang, Stephen Mussmann, Emma Pierson, Been
Kim, and Percy Liang.
\newblock Concept bottleneck models.
\newblock In \emph{International Conference on Machine Learning}, pp.\
5338--5348. PMLR, 2020.
\bibitem[McGrath et~al.(2022)McGrath, Kapishnikov, Toma{\v{s}}ev, Pearce,
Wattenberg, Hassabis, Kim, Paquet, and Kramnik]{mcgrath2022acquisition}
Thomas McGrath, Andrei Kapishnikov, Nenad Toma{\v{s}}ev, Adam Pearce, Martin
Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik.
\newblock Acquisition of chess knowledge in alphazero.
\newblock \emph{Proceedings of the National Academy of Sciences}, 119\penalty0
(47):\penalty0 e2206625119, 2022.
\bibitem[Olah et~al.(2020)Olah, Cammarata, Schubert, Goh, Petrov, and
Carter]{circuits}
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and
Shan Carter.
\newblock Zoom in: An introduction to circuits.
\newblock \emph{Distill}, 5, 03 2020.
\newblock \doi{10.23915/distill.00024.001}.
\bibitem[O'Mahony et~al.(2023)O'Mahony, Andrearczyk, M{\"u}ller, and
Graziani]{o2023disentangling}
Laura O'Mahony, Vincent Andrearczyk, Henning M{\"u}ller, and Mara Graziani.
\newblock Disentangling neuron representations with concept vectors.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition}, pp.\ 3769--3774, 2023.
\bibitem[Raghu et~al.(2017)Raghu, Gilmer, Yosinski, and
Sohl-Dickstein]{raghu2017svcca}
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein.
\newblock Svcca: Singular vector canonical correlation analysis for deep
learning dynamics and interpretability.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Rosenthal \& Fode(1963)Rosenthal and Fode]{experimenter_bias}
Robert Rosenthal and Kermit~L. Fode.
\newblock The effect of experimenter bias on the performance of the albino rat.
\newblock \emph{Behavioral Science}, 8\penalty0 (3):\penalty0 183--189, 1963.
\newblock \doi{https://doi.org/10.1002/bs.3830080302}.
\newblock URL
\url{https://onlinelibrary.wiley.com/doi/abs/10.1002/bs.3830080302}.
\bibitem[Rudin(2019)]{rudin2019stop}
Cynthia Rudin.
\newblock Stop explaining black box machine learning models for high stakes
decisions and use interpretable models instead.
\newblock \emph{Nature Machine Intelligence}, 1\penalty0 (5):\penalty0
206--215, 2019.
\bibitem[Rudin et~al.(2018)Rudin, Wang, and Coker]{rudin2018age}
Cynthia Rudin, Caroline Wang, and Beau Coker.
\newblock The age of secrecy and unfairness in recidivism prediction, 11 2018.
\bibitem[Russakovsky et~al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma,
Huang, Karpathy, Khosla, Bernstein, et~al.]{russakovsky2015imagenet}
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et~al.
\newblock Imagenet large scale visual recognition challenge.
\newblock \emph{International journal of computer vision}, 115\penalty0
(3):\penalty0 211--252, 2015.
\bibitem[Szegedy et~al.(2016)Szegedy, Vanhoucke, Ioffe, Shlens, and
Wojna]{szegedy2016rethinking}
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Wojna.
\newblock Rethinking the inception architecture for computer vision.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pp.\ 2818--2826, 2016.
\end{thebibliography}