Adversarial Examples Might be Avoidable: The Role of Data Concentration in Adversarial Robustness
Extracted source text ยท arXiv 2309.16096
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2022
% ready for submission
%\usepackage{neurips_2023}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
%\usepackage[preprint]{neurips_2023}
% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[final,nonatbib]{neurips_2023}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2022}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\hypersetup{
colorlinks,
linkcolor={red!50!black},
citecolor={blue!50!black},
urlcolor={blue!80!black}
}
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{amsmath,amssymb,amsthm}
\usepackage{thm-restate}
\usepackage[capitalise]{cleveref}
\usepackage{wrapfig, graphicx}
\usepackage{xargs}
\usepackage[colorinlistoftodos,prependcaption,textsize=tiny]{todonotes}
\usepackage{selectp}
\usepackage{caption,subcaption}
\usepackage{enumitem}
\usepackage{svg}
\def\R{\mathbb{R}}
\def\bbS{\mathbb{S}}
\DeclareMathOperator*{\bbP}{\mathbb{P}}
\def\cD{\mathcal{D}}
\def\cQ{\mathcal{Q}}
\def\cR{\mathcal{R}}
\def\cS{\mathcal{S}}
\def\cN{\mathcal{N}}
\def\cX{\mathcal{X}}
\def\cY{\mathcal{Y}}
\def\X{\mathbf{X}}
\def\Y{\mathbf{Y}}
\def\U{\mathbf{U}}
\def\V{\mathbf{V}}
\def\A{\mathbf{A}}
\def\Q{\mathbf{Q}}
\def\Z{\mathbf{Z}}
\def\I{\mathbf{I}}
\def\M{\mathbf{M}}
\def\N{\mathbf{N}}
\def\C{\mathbf{C}}
\def\W{\mathbf{W}}
\def\S{\mathbf{S}}
\def\T{\mathbf{T}}
\def\D{\mathbf{D}}
\DeclareMathOperator*{\bP}{\mathbf{P}}
\def\st{\ \ \text{s.t.} \ \ }
\def\a{\mathbf{a}}
\def\b{\mathbf{b}}
\def\c{c} % \def\c{\mathbf{c}}
\def\d{d} % \def\d{\mathbf{d}}
\def\e{e} % \def\e{\mathbf{e}}
\def\f{f} % \def\f{\mathbf{f}}
\def\m{\mathbf{m}}
\def\n{n}% \def\n{\mathbf{n}}
\def\p{\mathbf{p}}
\def\r{\mathbf{r}}
\def\s{s} % \def\s{\mathbf{s}}
\def\bt{t} % \def\bt{\mathbf{t}}
\def\u{\mathbf{u}}
\def\v{v} % \def\v{\mathbf{v}}
\def\w{\mathbf{w}}
\def\x{x} % \def\x{\mathbf{x}}
\def\y{\mathbf{y}}
\def\z{z} % \def\z{\mathbf{z}}
\def\ie{\textit{i.e.}}
\def\balpha{\mathbf{\alpha}}
\def\bPhi{\mathbf{\Phi}}
\newcommand{\myparagraph}[1]{\noindent\textbf{#1.}}
%\newcommand{\myparagraph}[1]{\noindent\textbf{#1.}}
\def\bmu{\boldsymbol{\mu}}
\def\balpha{\boldsymbol{\alpha}}
\def\Var{\mathbf{Var}}
\def\Covar{\mathbf{Covar}}
\DeclareMathOperator*\E{\mathbb{E}}
\newcommand{\F}{ {\rm F}}
\def\1{\mathbf{1}}
\def\0{\mathbf{0}}
\newcommand{\B}{\mathbf}
\newcommand{\Ber}{\text{Ber}}
%\newcommand{\eg}{\emph{e.g.}}
%\newcommand{\ie}{\rm i.e.}
\newcommand{\diag}[1]{{\rm diag}(#1)}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
\newcommand{\rom}[1]{%
\textup{\uppercase\expandafter{\romannumeral#1}}%
}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{definition}{Definition}[section]
\newtheorem{example}{Example}[section]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}
\crefformat{footnote}{#2\footnotemark[#1]#3}
%\long\def\rene#1{\footnote{\color{red}RV: #1}}
% \long\def\darshan#1{\footnote{\color{blue}DT: #1}}
%\long\def\ambar#1{\footnote{\color{teal}AP: #1}}
%\newcommandx{\darshan}[2][1=]{\todo[linecolor=blue,backgroundcolor=blue!25,bordercolor=blue,#1]{#2}}
\newcommandx{\ambartechnical}[2][1=]{\todo[linecolor=teal,backgroundcolor=teal!25,bordercolor=teal,#1]{AP Technical: #2}}
\newcommandx{\ambarwriting}[2][1=]{\todo[linecolor=orange,backgroundcolor=orange!25,bordercolor=orange,#1]{AP Writing: #2}}
% \newcommandx{\rene}[2][1=]{\todo[linecolor=red,backgroundcolor=red!25,bordercolor=red,#1]{RV: #2}}
\newcommand{\ambar}[1]{{\color{blue}AP:#1}}
\newcommand{\rene}[1]{{\color{orange}RV:#1}}
\def\apupdate#1{{\color{black}#1}}
% \def\apupdate#1{{\color{black}#1}}
% \title{Certified Robustness with \\ Low Dimensional Data Models}
% \title{Certified Robustness for Data in a \\ Union of Low Dimensional Subspaces}
% \title{On the Critical Role of Data Structure in \\ Adversarial Robustness}
% \title{Adversarial Examples Might be Avoidable: \\ The Critical Role of Data Structure in Adversarial Robustness}
\title{Adversarial Examples Might be Avoidable: The Role of Data Concentration in Adversarial Robustness}
% \title{The Role of Data Concentration in Adversarial Robustness: Adversarial Examples Might be Avoidable}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Ambar Pal \\
\texttt{ambar@jhu.edu} \And
Jeremias Sulam \\
\texttt{jsulam1@jhu.edu} \And
Ren\'e Vidal \\
\texttt{vidalr@upenn.edu}
}
%\outputonly{1-14}
%\outputonly{15-27}
\begin{document}
\maketitle
\begin{abstract}
The susceptibility of modern machine learning classifiers to adversarial examples has motivated theoretical results suggesting that these might be unavoidable. However, these results can be too general to be applicable to natural data distributions. Indeed, humans are quite robust for tasks involving vision. This apparent conflict motivates a deeper dive into the question: Are adversarial examples truly unavoidable?
In this work, we theoretically demonstrate that a key property of the data distribution -- concentration on small-volume subsets of the input space -- determines whether a robust classifier exists. We further demonstrate that, for a data distribution concentrated on a union of low-dimensional linear subspaces, utilizing structure in data naturally leads to classifiers that enjoy data-dependent polyhedral robustness guarantees, improving upon methods for provable certification in certain regimes.
%We further demonstrate that, for a data distribution concentrated on a union of low-dimensional linear subspaces, exploiting data structure naturally leads to classifiers that enjoy good robustness guarantees, improving upon methods for provable certification in certain regimes.
\end{abstract}
\section{Introduction, Motivation and Contributions} \label{sec:intro}
Research in adversarial learning has shown that traditional neural network based classification models are prone to anomalous behaviour when their inputs are modified by tiny, human-imperceptible perturbations. Such perturbations, called adversarial examples, lead to a large degradation in the accuracy of classifiers \cite{szegedy2013intriguing}. This behavior is problematic when such classification models are deployed in security sensitive applications. Accordingly, researchers have and continue to come up with \emph{defenses} against such adversarial attacks for neural networks.
Such defenses \cite{shafahi2019adversarial,wong2019fast,papernot2016distillation,guo2018countering} modify the training algorithm, alter the network weights, or employ preprocessing to obtain classifiers that have improved empirical performance against adversarially corrupted inputs. However, many of these defenses have been later broken by new adaptive attacks \cite{carliniA,carliniB}. This motivated recent impossibility results for adversarial defenses, which aim to show that all defenses admit adversarial examples. While initially such results were shown for specially parameterized data distributions \cite{fawzi}, they were subsequently expanded to cover general data distributions on the unit sphere and the unit cube \cite{goldstein}, as well as for distributions over more general manifolds \cite{dohmatob}. %\ambarwriting{Remember to mention the other results showing it might provably takes a lot of data to construct robust classifiers.}
On the other hand, we humans are an example of a classifier capable of very good (albeit imperfect \cite{humanadversarials}) robust accuracy against $\ell_2$-bounded attacks for natural image classification. Even more, a large body of recent work has constructed \emph{certified} defenses \cite{cohen2019certified,yang2020randomized,chiang2020certified,levine2020randomized,fischer2020certified,sulam2020adversarial} which obtain non-trivial performance guarantees under adversarially perturbed inputs for common datasets like MNIST, CIFAR-10 and ImageNet. This apparent contention between impossibility results and the existence of robust classifiers for natural datasets indicates that the bigger picture is more nuanced, and motivates a closer look at the impossibility results for adversarial examples.
% Motivated by this apparent conflict between existing impossibility results and the success of certified defenses on standard datasets, we take a closer look at the impossibility results for adversarial robustness.
Our first contribution is to show that these results can be circumvented by data distributions whose mass concentrates on small regions of the input space. This naturally leads to the question of whether such a construction is necessary for adversarial robustness. We answer this question in the affirmative, formally proving that a successful defense exists only when the data distribution concentrates on an exponentially small volume of the input space. At the same time, this suggests that exploiting the inherent structure in the data is critical for obtaining classifiers with broader robustness guarantees. %better robust classifiers.
Surprisingly, almost\footnote{See \cref{sec:relwork} for more details.} %\ambarwriting{Mention Soumayeh work on projected Randomized Smoothing, and maybe also Awasthi's!}
all \emph{certified} defenses do not exploit any structural aspects of the data distribution like concentration or low-dimensionality. %The few works that do exist in this direction \cite{awasthiA,B} work for the case when data lies on a single linear subspace.
Motivated by our theoretical findings, we study the special case of data distributions concentrated near a union of low-dimensional linear subspaces, to create a certified defense for perturbations that go beyond traditional $\ell_p$-norm bounds. We find that simply exploiting the low-dimensional data structure leads to a natural classification algorithm for which we can derive norm-independent polyhedral certificates. We show that our method can certify accurate predictions under adversarial examples with an $\ell_p$ norm larger than what can be certified by applying existing, off-the-shelf methods like randomized smoothing \cite{cohen2019certified}.
%We show that our method can be used to certify against adversarial perturbations having large $\ell_p$ norms, and hence applying an off-the-shelf method like randomized smoothing is unable to certify against such perturbations.
Thus, we demonstrate the importance of structure in data %\ambarwriting{Too many mentions of the vague term ``data-structure''} %
for both the theory and practice of certified adversarial robustness.
More precisely, we make the following main contributions in this work:
\begin{enumerate}
\item We formalize a notion of $(\epsilon, \delta)$-concentration of a probability distribution $q$ in \cref{sec:necessarycond}, which states that $q$ assigns at least $1 - \delta$ mass to a subset of the ambient space having volume $O(\exp{(-n \epsilon)})$. We show that $(\epsilon, \delta)$-concentration of $q$ is a necessary condition for the existence of any classifier obtaining at most $\delta$ error over $q$, under perturbations of size $\epsilon$.
\item We find that $(\epsilon, \delta)$-concentration is too general to be a sufficient condition for the existence of a robust classifier, and we follow up with a stronger notion of concentration in \cref{sec:sufficientcond} which is sufficient for the existence of robust classifiers. Following this stronger notion, we construct an example of a strongly-concentrated distribution, %\ambartechnical{extend to the unit cube}
which circumvents existing impossibility results on the existence of robust classifiers.
\item We then consider a data distribution $q$ concentrated on a union of low-dimensional linear subspaces in \cref{sec:subspace}.
%For this distribution, we construct a classifier robust to on-subspace perturbations, and obtain its associated certificates in \cref{sec:onsubspace}. We demonstrate that the certified regions have a simple description as the convex conic hull of selected points in the training data.
%\item Finally, we consider a data distribution concentrated on a union of low-dimensional linear subspaces perturbed by additive $\ell_2$ bounded noise in \cref{sec:offsubspace}.
%For this distribution, we construct a classifier robust to (a subset of) perturbations bounded by the $\ell_2$ distance from the underlying subspaces, and obtain its associated certificates.
% For this distribution,
We construct a classifier for $q$ that is robust to perturbations following threat models more general than $\ell_p$. Our analysis results in polyhedral %under more general threat models, resulting in attacks that are constrained to polyhedrons %Note that the adversarial attacks might not have small $\ell_p$ norm themselves, specifically, this is a superset of the commonly studied $\ell_2$ attack model. We demonstrate that our
certified regions whose faces and extreme rays are described by selected points in the training data.
\item We perform empirical evaluations on MNIST in \cref{sec:expt}, demonstrating that our certificates are complementary to existing off-the-shelf approaches like Randomized Smoothing (RS), in the sense that
both methods have different strengths. In particular, we demonstrate a region of adversarial perturbations where our method is certifiably robust, but RS is not. We then combine our method with RS to obtain certificates that enjoy the best of both worlds.% roneither is \emph{better} than the other. In particular, we demonstrate that there is a set of adversarial perturbations that our method is robust to but RS is not. Conversely, we also find perturbations that are certified by RS but not by our method. \ambartechnical{Empirically combine RS and our method to obtain the \emph{union} of these methods?}
\end{enumerate}
%
% The remainder of this manuscript is organized as follows. In \cref{sec:necessarycond}, we introduce notation, define concentration and show that it is a neccesary condition for robustness. We follow up in \cref{sec:sufficientcond} to show that a stronger notion of concentration is sufficient for robustness. Then, we study a particular distribution concentrated over a union of linear subspaces in \cref{sec:subspace} and derive certificates of robustness, which are then evaluated empirically in \cref{sec:expt}.
% Finally, we note connections to past literature and conclude in \cref{sec:conclusion}.
%
\section{Existence of Robust Classifier Implies Concentration} \label{sec:necessarycond}
We will consider a classification problem over $\cX \times \cY$ defined by the data distribution $p$ such that $\cX$ is bounded and $\cY = \{1, 2, \ldots, K\}$. We let $q_k$ % \colon \cX \to \R_{\geq 0}$
denote the conditional distribution $p_{X|Y=k}$ for class $k \in \cY$. We will assume that the data is normalized, i.e., $\cX = B_{\ell_2}(0, 1)$, and the boundary of the domain is far from the data, i.e., for any $x \sim q_k$, an adversarial perturbation of $\ell_2$ norm at most $\epsilon$ does not take $x$ outside the domain $\cX$.\footnote{More details in \cref{proof:concentration}.}
% Previous Version
% We will consider a classification problem over $\cX \times \cY$ defined by the data distribution $p$ such that $\cX$ is bounded and $\cY = \{1, 2, \ldots, K\}$. WLOG, assume that all quantities are normalized so that $\cX$ is an $\ell_2$ ball of radius $1$, i.e., $\cX = B_{\ell_2}(0, 1)$. Let $q_k \colon \cX \to \R_{\geq 0}$ denote the conditional distribution for class $k \in \cY$, having support $Q_k$. We will impose a technical restriction \ambarwriting{I wonder whether this should be fleshed out completely here, or moved to the appendix. A casual reader might take this $\epsilon \leq 0.1$ to be evidence that our results are only true for low $\epsilon$.} to ensure that an adversarial perturbation on any data-point sampled from $p$ does not take the resultant attacked point outside the domain $\cX$. For this, we will assume that there is a sufficient gap between the supports and the boundary of the domain, i.e., $Q_k \subseteq B_{\ell_2}(0, 0.9)$, and that the $\ell_2$--adversarial attack has strength $\epsilon \leq 0.1$. The numerical values are arbitrary here, and taken to simplify notation. \ambartechnical{Provide the more complicated proof without this assumption, and state how the concentration criterion changes to a much less interpretable version}
We define the robust risk of a classifier $f \colon \cX \to \cY$ against an adversary making perturbations whose $\ell_2$ norm is bounded by $\epsilon$ as \footnote{Note that for $f(\bar x)$ to be defined, it is implicit that $\bar x \in \cX$ in \eqref{robustness-defn}.}
\begin{equation}
R(f, \epsilon) = \Pr_{(x, y) \sim p} \left( \exists \bar x \in B_{\ell_2}(x, \epsilon) \text{ such that } f(\bar x) \neq y \right). \label{robustness-defn}
\end{equation}
% To begin discussion about the existence of robust classifiers, we have to firstly define a robust classifier.
We can now define a robust classifier in our setting.
\begin{definition}[Robust Classifier]
A classifier $g$ is defined to be $(\epsilon, \delta)$-robust if the robust risk against perturbations with $\ell_2$ norm bounded by $\epsilon$ is at most $\delta$, \ie, if $R(g, \epsilon) \leq \delta$. %Here, $\delta$ is to be thought of as close to $0$.
\end{definition}
The goal of this section is to show that if our data distribution $p$ admits an $(\epsilon, \delta)$--robust classifier, then $p$ has to be \emph{concentrated}.
%Given that our data distribution $p$ admits an $(\epsilon, \delta)$--robust classifier $f$, the purpose of this section is now to show that $p$ is \emph{concentrated}.
Intuitively, this means that $p$ assigns a ``large'' measure to sets of ``small'' volume. We define this formally now.
\begin{definition}[Concentrated Distribution] \label{def:conc}
A probability distribution $q$ over a domain $\cX \subseteq \R^n$ is said to be $(C,\epsilon, \delta)$-concentrated, if there exists a subset $S \subseteq \cX$ such that $q(S) \geq 1 - \delta$ but ${\rm Vol}(S) \leq C \exp(-n \epsilon)$. % for some constant $C > 0$.
Here, %$\delta$ is thought of close to $0$, and
${\rm Vol}$ denotes the standard Lebesgue measure on $\R^n$, and $q(S)$ denotes the measure of
%probability of sampling a point in
$S$ under $q$.
\end{definition}
With the above definitions in place, we are ready to state our first main result.
\begin{restatable}[]{theorem}{concentration}
If there exists an $(\epsilon, \delta)$-robust classifier $f$ for a data distribution $p$, then at least one of the class conditionals $q_1, q_2, \ldots, q_K$, say $q_{\bar k}$, must be $(\bar C, \epsilon, \delta)$--concentrated. Further, if the classes are balanced, then all the class conditionals are $(C_{\max}, \epsilon, K \delta)$-concentrated. Here, $\bar C = {\rm Vol}\{x \colon f(x) = \bar k\}$, and $C_{\max} = \max_k {\rm Vol}\{x \colon f(x) = k\}$ are constants dependent on $f$.
\label{th:concentration}
\end{restatable}
The proof is a natural application of the Brunn-Minkowski theorem from high-dimensional geometry, essentially using the fact that an $\epsilon$-shrinkage of a high-dimensional set has very small volume. We provide a brief sketch here, deferring the full proof to \cref{proof:concentration}.
%The proof utilizes the Brunn-Minkowski theorem from high-dimensional geometry, and we provide a brief sketch here, deferring the full proof to \cref{proof:concentration}.
\begin{proof}[Proof Sketch]
% \proof[Proof Sketch]
Due to the existence of a robust classifier $f$, \ie, $R(f, \epsilon) \leq \delta$, the first observation is that there must be at least one class which is classified with robust accuracy at least $1 - \delta$. Say this class is $k$, and the set of all points which do not admit an $\epsilon$-adversarial example for class $k$ is $S$. Now, the second step is to show that $S$ has the same measure (under $q_k$) as the $\epsilon$-shrinkage (in the $\ell_2$ norm) of the set of all points classified as class $k$. Finally, the third step involves using the Brunn-Minkowski theorem, to show that this $\epsilon$-shrinkage has a volume $O(\exp(-n \epsilon))$, thus completing the argument.
% Due to the existence of a robust classifier $f$, \ie, $R(f, \epsilon) \leq \delta$, the first observation is that there exists at least one subset $S$ of the input space, and a corresponding class $k$, such that, all points in $S$ are classified to be in class $k$ with accuracy at least $1 - \delta$ (under adversarial perturbations of $\ell_2$ norm at most $\epsilon$). The second step then involves showing that this region has the same measure as an $\epsilon$--shrinkage (in the $\ell_2$ norm) of the set of all points that are assigned the class $k$. Finally, the third step involves using the Brunn-Minkowski theorem, to show that this $\epsilon$-shrinkage has a volume $O(\exp(-n \epsilon))$, thus completing the argument.
% \endproof
\end{proof}
% \proof[Proof Sketch] Due to the existence of a robust classifier $f$, i.e. $R(f, \epsilon) \leq \delta$, the first observation is that there exists at least one subset of the input space whose samples are classified accurately under adversarial perturbations of $\ell_2$ norm at most $\epsilon$ with measure at least $1 - \delta$ under the class conditional $q_k$, for some class $k$. The second step then involves showing that this region has the same measure as an $\epsilon$--shrinkage (in the $\ell_2$ norm) of the set of all points that are assigned the class $k$ by $f$. Finally, the third step involves using the Brunn-Minkowski inequality, to show that this $\epsilon$--shrinkage has a volume $O(\exp(-n \epsilon))$, thus completing the argument.
\myparagraph{Discussion on \cref{th:concentration}} We pause here to understand some implications of this result.
\begin{itemize}[leftmargin=*]
\item Firstly, recall the apparently conflicting conclusions from \cref{sec:intro} between impossibility results (suggesting that robust classifiers do not exist) and the existence of robust classifiers in practice (such as that of human vision for natural data distributions). \cref{th:concentration} shows that whenever a robust classifier exists, the underlying data distribution has to be concentrated. In particular, this suggests that natural distributions corresponding to MNIST, CIFAR and ImageNet might be concentrated. This indicates a resolution to the conflict: concentrated distributions must somehow circumvent existing impossibility results. Indeed, this is precisely what we will show in \cref{sec:sufficientcond}.
\item Secondly, while our results are derived for the $\ell_2$ norm, it is not very hard to extend this reasoning to general $\ell_p$ norms. In other words, whenever a classifier robust to $\ell_p$-norm perturbations exists, the underlying data distribution must be concentrated.% w.r.t. the corresponding $\ell_p$ distance.
\item Thirdly, \cref{th:concentration} has a direct implication towards classifier design. Since we now know that natural image distributions are concentrated, one should design classifiers that are tuned for small-volume regions in the input space. This might be the deeper principle behind the recent success \cite{zhang2022rethinking} of robust classifiers adapted to $\ell_p$-ball like regions in the input space.
\item Finally, the \emph{extent} of concentration implied by \cref{th:concentration} depends on the classifier $f$, via the parameters $\epsilon, \delta$ and $\bar C$. On one hand, we get \emph{high} concentration when $\epsilon$ is large, $\delta$ is small, and $\bar C$ is small. On the other hand, if the distribution $p$ admits an $(\epsilon, \delta)$-robust classifier such that $\bar C$ is large (e.g., a constant classifier), then we get \emph{low} concentration via \cref{th:concentration}. This is not a limitation of our proof technique, but a consequence of the fact that some simple data distributions might be very lowly concentrated, but still admit very robust classifiers, e.g., for a distribution having $95\%$ dogs and $5\%$ cats, the constant classifier which always predicts ``dog'' is quite robust.
\end{itemize}
%\ambarwriting{Write a paragraph on implications, showing how existence of humans shows that natural image datasets are concentrated in $\ell_2$. Comment how this can be easily extended to $\ell_p$. Then talk about how this provides theoretical justification for recent works on certified $\ell_\infty$ robustness which end up at classifiers respecting $\ell_\infty$ concentration.}
We have thus seen that data concentration is a necessary condition for the existence of a robust classifier. A natural question is whether it is also sufficient. We address this question now.%in the next section.
%
\section{Strong Concentration Implies Existence of Robust Classifier} \label{sec:sufficientcond}
Say our distribution $p$ is such that all the class conditionals $q_1, q_2, \ldots, q_k$ are $(C, \epsilon, \delta)$-concentrated. Is this sufficient for the existence of a robust classifier? The answer is negative, as we have not
% {\color{red}Theorem \ref{th:concentration} tells us that if there exists an $(\epsilon, \delta)$-robust classifier, then at least one of the class conditionals $q_1, q_2, \ldots, q_K$ must be $(\epsilon, \delta)$-concentrated.}
% %The definition of $(\epsilon, \delta)$-concentration tells us that each of the class conditionals $q_1, q_2, \ldots, q_K$ are concentrated.
% However, this does not
precluded the case where all of the $q_k$ are concentrated over the same subset $S$ of the ambient space. In other words, it might be possible that there exists a small-volume set $S \subseteq \cX$ such that $q_k(S)$ is high for all $k$. This means that whenever a data point lies in $S$, it would be hard to distinguish which class it came from. In this case, %it might be impossible to distinguish between the classes, and
even an accurate classifier might not exist, let alone a robust classifier\footnote{Recall that classifier not accurate at a point $(x, y)$, i.e., $f(x) \neq y$, is by definition not robust at $x$, as a $v=0$ perturbation is already sufficient to ensure $f(x + v) \neq y$.}. To get around such issues, we define a stronger notion of concentration, as follows.
\begin{definition}[Strongly Concentrated Distributions] \label{def:strongconc} A distribution $p$ is said to be $(\epsilon, \delta, \gamma)$-strongly-concentrated if each class conditional distribution $q_k$ is
concentrated over the set $S_k \subseteq \cX$ such that $q_k(S_k) \geq 1 - \delta$, and $q_k\left(\bigcup_{k' \neq k} S_{k'}^{+2\epsilon}\right) \leq \gamma$, where $S^{+\epsilon}$ denotes the $\epsilon$-expansion of the set $S$ in the $\ell_2$ norm, i.e., $S^{+\epsilon} = \{ x \colon \exists \bar x \in S \text{ such that } \|x - \bar x\|_2 \leq \epsilon\}$.
\end{definition}
In essence, \cref{def:strongconc} states that each of the class conditionals are concentrated on subsets of the ambient space, which do not intersect too much with one another\footnote{More detailed discussion in \cref{app:strongconcdisc}.}. Hence, it is natural to expect that we would be able to construct a robust classifier by exploiting these subsets. Building upon this idea, we are able to show \cref{th:strongconc}:
\begin{restatable}[]{theorem}{strongconc}
% \begin{theorem} \label{th:strongconc}
If the data distribution $p$ is
%such that the set of class conditionals $\cQ = \{q_1, q_2, \ldots, q_k\}$ is
$(\epsilon, \delta, \gamma)$-strongly-concentrated, then there exists an $(\epsilon, \delta + \gamma)$-robust classifier for $p$. \label{th:strongconc}
% \end{theorem}
\end{restatable}
%
The basic observation behind this result is that if the conditional distributions $q_k$ had disjoint supports which were well-separated from each other, then one could obtain a robust classifier by predicting the class $k$ on the entire $\epsilon$-expansion of the set $S_k$ where the conditional $q_k$ concentrates, for all $k$. To go beyond this idealized case, we can exploit the strong concentration condition to carefully remove the intersections at the cost of at most $\gamma$ in robust accuracy. We make these arguments more precise in the full proof, deferred to \cref{proof:strongconc}, and we pause here to note some implications for existing results.
%
% The basic argument behind this result is to observe that, if the conditional distributions $q_k$ had disjoint supports, one could obtain a robust classifier by predicting the class $k$ on the entire $\epsilon$--expansion of the set $S_k$ where the conditional $q_k$ concentrates, for all $k$. To go beyond this simplistic case, we can exploit the strong concentration condition to carefully remove the intersections at the cost of at most $\gamma$ in robust accuracy. We make these arguments more precise in the full proof, deferred to \cref{proof:strongconc}, and we pause here to note some implications for existing results.
\myparagraph{Implications for Existing Impossibility Results} %It is useful to first understand the result in \cref{th:strongconc} in the context of the impossibility results for robust classifiers referred in \cref{sec:intro}.
% We will provide an example of how the ideas developed in \cref{th:strongconc} extend to other metric spaces, and provide a way to circumvent existing impossibility results in adversarial robustness \cite{goldstein}.
To understand how \cref{th:strongconc} circumvents the previous impossibility results, consider the setting from \cite{goldstein} where the data domain is the sphere $\bbS^{n-1} = \{x \in \R^n \colon \|x\|_2 = 1\}$, and we have a binary classification setting with class conditionals $q_1$ and $q_2$. The adversary is allowed to make bounded perturbations w.r.t. the geodesic distance. %a data distribution on the unit sphere which circumvents existing impossibility results on adversarial robustness, by exploiting strong concentration of the data distribution, with the metric taken to be the geodesic distance on the sphere.
% Previous version
% While we have derived our results thus far considering the domain $\cX$ to be $B_{\ell_2}(0, 1)$ endowed with the euclidean distance, the same principles can be applied when the domain (and consequently, the adversarial perturbations and the supports of the conditional densities) is not full-dimensional or euclidean, and the $\epsilon$--expansion is w.r.t. a metric on $\cX$. We provide an example of a strongly-concentrated %%%%
% \ambartechnical{Should we be more precise here about the $(\epsilon, \delta, \gamma)$ parameters of the strong concentration? I did not because $\epsilon$--expansion would be now defined w.r.t to a general distance function on $\cX$ which is not the euclidean $\ell_2$ distance.} %%%%%
% data distribution on the unit sphere $\cX = \bbS^{n-1} = \{x \in \R^n \colon \|x\|_2 = 1\}$ endowed with the geodesic distance in the binary classification setting, which circumvents existing impossibility results on adversarial robustness by exploiting strong concentration of the data.
In this setting, it can be shown (see \cite[Theorem 1]{goldstein}) that any classifier admits $\epsilon$-adversarial examples for the minority class (say class $1$), with probability at least
\begin{equation}
1 - \alpha_{q_1} \beta \exp \left(-\frac{n - 1}{2} \epsilon^2\right) ,\label{eq:geodesiclb}
\end{equation}
% \begin{equation}
% 1 - \left(\sup_{x \in \bbS^{n-1}} q_1(x)\right) \cdot \mu_{n-1}(\bbS^{n-1}) \cdot \exp \left(-\frac{n - 1}{2} \epsilon^2\right) \left(\frac{\pi}{8}\right)^{1/2},\label{eq:geodesiclb}
% \end{equation}
where $\alpha_{q_1} = \sup_{x \in \bbS^{n-1}} q_1(x)$ depends on the conditional distribution $q_1$, and $\beta$ is a normalizing constant that depends on the dimension $n$. %$\beta = \mu_{n-1}(\bbS^{n-1}) \cdot \sqrt{\pi/8}$ is a normalizing constant. Here, $\mu_{n-1}$ denotes the $(n-1)$--dimensional volume (surface-area).
Note that this result assumes little about the conditional $q_1$. Now, by constructing a strongly-concentrated data distribution over the domain, we will show that the
\begin{wrapfigure}{r}{0.3\textwidth}
% \vspace{-1ex}
\centering
\includegraphics[trim={3cm 3cm 1cm 3cm},clip,width=0.3\textwidth]{fig/page_2.png}
\caption{A plot of $q_1$. Redder colors denote a larger density, and the gray plane denotes the robust classifier.} \label{fig:escape1}
\vspace{-6ex}
\end{wrapfigure}
lower bound in \eqref{eq:geodesiclb} becomes vacuous.
%\begin{restatable}[]{example}{escape}
\begin{example}
\label{escape}
The data domain is the unit sphere $\bbS^{n-1}$ equipped with the geodesic distance $d$. The label domain is $\{1, 2\}$. $P$ is an arbitrary, but fixed, point lying on $\bbS^{n-1}$. The conditional density of class $1$, i.e., $q_1$ is now defined as
\begin{gather*}
q_1(x) = \begin{cases}
\frac{1}{C} \frac{1}{\sin^{n-2} d(x, P)}, &\text{ if } d(x, P) \leq 0.1 \\
0, &\text{ otherwise }
\end{cases},
\end{gather*}
where $C = 0.1$ is a normalizing constant. The conditional density of class 2 is defined to be uniform over the complement of the support of $q_1$, i.e. $q_2 = {\rm Unif}(\{x \in \bbS^{n-1} \colon d(x, P) > 0.1\})$. Finally, the classes are balanced, i.e., $p_Y(1)= p_Y(2) = 1/2$.
\end{example}
%\end{restatable}
The data distribution constructed in \cref{escape} makes \cref{eq:geodesiclb} vacuous, as the supremum over the density $q_1$ is unbounded. %Further, as the distribution is strongly-concentrated, a robust classifier should exist.
Additionally, the linear classifier defined by the half-space $\{x \colon \langle x, P \rangle \leq \cos(0.1)\}$ is robust (\cref{app:escape} provides a derivation of the robust risk, and further comments on generalizing this example). \cref{escape} is plotted for $n = 3$ dimensions in \cref{fig:escape1}.%{\color{teal}AP: Write proof in Appendix and link to it.}
% \ambartechnical{This paragraph is extremely vague. Add details like parameters of the strong concentration, ``should exist''? but the results earlier were for euclidean distance, parameters of robustness}
% \begin{figure}
% % \begin{restatable}[]{figure}{examples}
% \centering
% \includegraphics[width=0.49\textwidth]{fig/example1}
% \includegraphics[width=0.49\textwidth]{fig/subspace_conc.eps}
% \caption{Left: A plot of $q_1$ from \cref{escape} in dimension $n = 3$. Redder colors denote a larger density. The gray half-space denotes the robust classifier. Right: A plot of $q_1$(orange) and $q_2$(violet) from \cref{ex:subspaceconc} in dimension $n = 2$, along with the decision regions of the robust classifier $f$. } \label{fig:escape}
% \end{figure}
% \end{restatable}
\paragraph{Compatibility of \cref{th:strongconc} with existing Negative Results} Thus, we see that strongly concentrated distributions are able to circumvent existing impossibility results on the existence of robust classifiers. However, this does \emph{not} invalidate any existing results. Firstly, measure-concentration-based results \cite{goldstein,fawzi,dohmatob} provide non-vacuous guarantees given a \emph{sufficiently flat} (not concentrated) data distribution, and hence do not contradict our results. Secondly, our results are existential and do not provide, in general, an algorithm to \emph{construct} a robust classifier given a strongly-concentrated distribution. Hence, we also do not contradict the existing stream of results on the computational hardness of finding robust classifiers \cite{bubeck,madryA,madryB}. Our positive results are complementary to all such negative results, demonstrating a general class of data distributions where robust classifiers do exist.
For the reminder of this paper, we will look at a specific member of the above class of strongly concentrated data distributions and show how we can practically construct robust classifiers.
\section{Adversarially Robust Classification on Union of Linear Subspaces} \label{sec:subspace}
The union of subspaces model has been shown to be very useful in classical computer vision for a wide variety of tasks, which include clustering faces under varying illumination, image segmentation, and video segmentation \cite{renebook}.
Its concise mathematical description often enables the construction and theoretical analysis of algorithms that also perform well in practice. In this section, we will study robust classification on data distributions concentrated on a union of low-dimensional linear subspaces. This data structure will allow us to obtain a non-trivial, practically relevant case where we can show a provable improvement over existing methods for constructing robust classifiers in certain settings. Before delving further, we now provide a simple example (which is illustrated in \cref{fig:escape2}) demonstrating how distributions concentrated about linear subspaces are concentrated precisely in the sense of \cref{def:strongconc}, and therefore allow for the existence of adversarially robust classifiers.
\begin{wrapfigure}{r}{0.3\textwidth}
\vspace{1ex}
\centering
\includegraphics[width=0.3\textwidth]{fig/subspace_conc_simple_recolor}
\caption{A plot of $q_1$ (orange), $q_2$ (violet) and the decision boundaries of $f$ (dashed).}
\label{fig:escape2}
\vspace{-7.5ex}
\end{wrapfigure}
\begin{restatable}[]{example}{subspaceconc}
%\begin{example}
The data domain is the ball $B_{\ell_\infty}(0, 1)$ %$ = \{x \colon \|x\|_\infty \leq 1\}$
equipped with the $\ell_2$ distance. The label domain is $\{1, 2\}$. Subspace $S_1$ is given by $S_1 = \{x \colon x^\top e_1 = 0\}$, and $S_2$ is given by $S_2 = \{x \colon x^\top e_2 = 0\}$, where $e_1, e_2$ are the standard unit vectors. The conditional densities are defined as
\begin{align*}
q_1 &= {\rm Unif}(\{x \colon \|x\|_\infty \leq 1, |x^\top e_1| \leq e^{-\alpha}/2\}), \text{ and, } \\
q_2 &= {\rm Unif}(\{x \colon \|x\|_\infty \leq 1, |x^\top e_2| \leq e^{-\alpha}/2\}),
\end{align*}
where $\alpha>0$ is a large constant. Finally, the classes are balanced, \ie, $p_Y(1) = p_Y(2) = 1/2$. %
With these parameters, $q_1, q_2$ are both $(0.5,\alpha / n - 1, 0)$-concentrated over their respective supports. %$S_1^{+{e^{-\alpha}}} \cap B_{\ell_\infty}(0, 1)$,
%and similarly $q_2$ is $(\alpha, 0)$--concentrated over its support., %$S_2^{+{e^{-\alpha}}} \cap B_{\ell_\infty}(0, 1)$.
Additionally, %the set $\cQ = \{q_1, q_2\}$
$p$ is $(\epsilon, 0, e^{-\alpha}/2 + 2\epsilon)$--strongly-concentrated. %\ambartechnical{There might be a 0.5 constant off here}.
A robust classifier $f$ can be constructed following the proof of \cref{th:strongconc}, and it obtains a robust accuracy $R(f, \epsilon) \leq e^{-\alpha}/2 + 2\epsilon$. See \cref{app:subspaceconc} for more details. \label{ex:subspaceconc}
%\end{example}
\end{restatable}
We will now study a specific choice of $p$ that generalizes \cref{ex:subspaceconc} and will let us move beyond the above simple binary setting. Recall that we have a classification problem specified by a data distribution $p$ over the data domain $\cX \times \cY = B(0, 1) \times \{1, 2, \ldots, K\}$. Firstly, the classes are balanced, \ie, $p_Y(k) = 1 / K$ for all $k \in \cY$. Secondly, %for all $k \in \cY$,
the conditional density, i.e., $q_k = p_{X | Y = k}$, is concentrated %\ambartechnical{Have to derive the parameters of concentration, and specify $q_k$ here}
on the set $S^{+\gamma}_k \cap \cX$, where $S_k$ is a low-dimensional linear subspace, and the superscript denotes an $\gamma$-expansion, for a small $\gamma > 0$.%Thirdly, for simplicity, we will additionally assume that $q_k(S_k \cap S_{k'} \cap \cX) = 0$ for all pairs $(k, k'), k' \neq k$.
For the purpose of building our robust classifier, we will assume access to a training dataset of $M$ \emph{clean} data points $(\s_1, y_1), (\s_2, y_2), \ldots, (\s_M, y_M)$, such that, for all $i$, the point $\s_i$ lies exactly on one of the $K$ low-dimensional linear subspaces. We will use the notation $\S = [\s_1, \s_2, \ldots, \s_M]$ for the training data matrix and $\y = (y_1, y_2, \ldots, y_M)$ for the training labels. We will assume that $M$ is large enough that every $\x \in \cup_k S_k$ can be represented as a linear combination of the columns of $\S$.
% Then, given $\{(\s_i,y_i) \in \cX \times \cY\}_{i=1}^M$, and a data-point $\x \sim p$,
Now, the robust classification problem we aim to tackle is to obtain a predictor $g \colon \cX \to \cY$ which %predicts the label $g(\x)$ that
obtains a low robust risk, with respect to an additive adversary $\mathcal{A}$ that we now define. For any data-point $\x \sim p$, $\mathcal{A}$ will be constrained to make an additive perturbation $v$ such that ${\rm dist}_{\ell_2}(\x + \v, \cup_i S_i) \leq \epsilon$. In other words, the attacked point can have $\ell_2$ distance at most $\epsilon$ from any of the linear subspaces $S_1, \ldots, S_k$. Note that $\mathcal{A}$ is more powerful than an $\ell_2$-bounded adversary as the norm of the perturbation $\|\v\|_2$ might be large, as $\v$ might be parallel to a subspace. %{\color{orange} This is illustrated in \cref{fig:attackmodel}.} \ambarwriting{Make this figure.}
Under such an adversary $\mathcal{A}$, given a (possibly adversarially perturbed) input $\x$, it makes sense to try to recover the corresponding point $\s$ lying on the union of subspaces, such that $\x = \s + \n$, such that $\|\n\|_2 \leq \epsilon$. One way to do this is to represent $\s$ as
% Given a (possibly adversarially perturbed) input $\x$, we propose to construct our robust classifier by obtaining $\c$ such that the projected, on-subspace counterpart $\s$ can be represented as
a linear combination of a small number of columns of $\S$, \ie, $\x = \S \c + \n$.
% We are also given a data-point $\x \in \bar \cX^\epsilon$, and predict the label in a manner that is robust to.
%
%Similar to \cref{sec:noiseless}, we are now given \emph{un-perturbed} data-points $(\s_1, y_1), (\s_2, y_2), \ldots, (\s_M, y_M)$\rene{$\s_i$ vs $s_i$}\rene{I'm not liking data are noiseless. AP: It is natural to have a training set comprising of noiseless inputs for developing a classifier, and at test time be expected to be robust to noisy (adversarially attacked) inputs. I will clarify this. RV: That would certainly help, but it is still not my point. One thing is noisy data, another is poisoned or adversarially attacked data. AP: I agree, I removed the noisy terminology since it clashes with what people in the clustering literature consider to be the noisy case. I use out-of-subspace terminology now. I still need to define the data-domain using noise, because otherwise the label function is undefined outside the subspace. I also stress everywhere that we assume access to a clean, noiseless training data-set.} such that $\s_i \in S_1 \cup \ldots \cup S_K$. We are also given a noisy test point $\x$, and our goal is again to obtain the linear subspace $S_y$ that its denoised point
%s generating point
%\rene{now the subspaces are generated by a point? AP: Indeed, this was confusing. Fixed.}
%belongs to, in the sense $\x = \s + \n$, $\s \in S_y, \n \in B_\epsilon$. We propose to construct our robust classifier by obtaining $\c$ such that $\s$ can be represented as a linear combination of a small number of columns of $\S$, \ie, $\x = \S \c + \n$, where $\S = [\s_1, \s_2, \ldots, \s_M]$. Then we would predict the majority class, as $\texttt{Majority}(\{y_i \colon i \in {\rm supp}(\c)\})$.
%
This can be formulated as an optimization problem that minimizes the cardinality of $c$, given by $\|c\|_0$, subject to an approximation error constraint. Since such a problem is hard because of the $\ell_0$ pseudo-norm, we relax this to the problem
% \begin{equation*}
% \min_\c \| \c \|_0 ~~\text{ s.t. }~~ \|\x - \S \c\|_2 \leq \epsilon \label{eq:noisel0}.
% \end{equation*}
% The above problem is hard due to the $\ell_0$ objective, and we relax it to
\begin{equation}
\min_\c \| \c \|_1 ~~\text{s.t.} ~~\|\x - \S \c\|_2 \leq \epsilon \label{eq:primalnoiseraw}.
\end{equation}
Under a suitable choice of $\lambda$, this problem can be equivalently written as%
%\footnote{Note that \eqref{eq:primalnoiseraw} is equivalent to \eqref{eq:primalnoise} under a suitable multiplier $\lambda$.} as %\rene{The equivalent formulation is one where $\|e\|\leq \epsilon$. This is either a relaxed formulation, or may be equivalent if $\lambda$ is the optimal dual. AP: True, I skipped this reasoning as it is basic optimization. I will include this as a footnote, since it is causing confusion.}
\begin{equation}
\min_{\c, \e} \| \c \|_1 + \frac{\lambda}{2} \|\e\|_2^2 ~~\text{s.t.} ~~\x = \S \c + \e \label{eq:primalnoise},
\end{equation}
for which we can obtain the dual problem given by
\begin{equation}
\max_{\d} \langle \x, \d \rangle - \frac{1}{2 \lambda} \|\d\|_2^2 ~~\text{s.t.} ~~ \|\S^\top \d \|_\infty \leq 1 \label{eq:dualnoise}.
\end{equation}
% \paragraph{High Level Idea}
Our main observation is to leverage the stability of the set of active constraints of this dual to obtain a robust classifier. One can note that each constraint of \cref{eq:dualnoise} corresponds to one training data point $s_i$ -- when the $i^{\rm th}$ constraint is active at optimality, $s_i$ is being used to reconstruct $x$. Intuitively, one should then somehow use the label $y_i$ while predicting the label for $x$. Indeed, we will show that predicting the majority label among the active $y_i$ leads to a robust classifier.
We will firstly obtain a geometric characterization of the problem in \cref{eq:dualnoise} by viewing it as the evaluation of a projection operator onto a certain convex set, illustrated in \cref{fig:geom2}. % This view will allow us to obtain a suitable dual classifier and the corresponding certificates.%\ambar{Compare this figure to the analysis of subspace classification, and have a discussion where one can see how we transition from everything in the support should be correct, to majority in the support should be correct. Done.}
%
Observe that for $\lambda > 0$, the objective \eqref{eq:dualnoise} is strongly concave in $\d$ and the problem has a unique solution, denoted by $\d^*_\lambda(\x)$.
% We rewrite \eqref{eq:dualnoise} as
% \begin{equation*}
% \max_{\d} \frac{1}{2\lambda} \|\lambda \x\|_2^2 - \frac{1}{2\lambda} \left( \|\lambda \x\|_2^2 - 2 \langle \lambda \x, \d \rangle + \|\d\|_2^2 \right) ~~\text{ s.t. } ~~\|\S^\top \d \|_\infty \leq 1.
% \end{equation*}
% Ignoring constants, and multiplying throughout by $-2 \lambda$ we have
It is not hard to show that this solution can be obtained by the projection operator
\begin{equation}
\d^*_\lambda(\x) = \left( \argmin_{\d} \|\lambda \x - \d\|_2 ~~\text{s.t. }~~ \|\S^\top \d \|_\infty \leq 1 \right) = {\rm Proj}_{K^\circ}(\lambda \x), \label{eq:optisproj}
\end{equation}
where $K^\circ$ is the polar of the convex hull of $\pm \S$. Denoting $\T = [\S, -\S]$, we can rewrite Problem \eqref{eq:optisproj} as $\d^*_\lambda(\x) = \left( \argmin_{\d} \|\lambda \x - \d\|_2 \text{ sub. to } \T^\top \d \leq \1 \right)$. We now define the set of active constraints as
\begin{equation}
A_\lambda(\x) = \{\bt_i \colon \langle \bt_i, \d^*_{\lambda}(\x) \rangle = 1\}. \label{eq:activelambda}
\end{equation}
\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-3ex}
\centering
\includegraphics[width=0.4\textwidth]{fig/geom2_small}
\caption{Geometry of the dual problem \eqref{eq:dualnoise}. See description on the left.}
\label{fig:geom2}
\vspace{-3ex}
\end{wrapfigure}
%
\myparagraph{Geometry of the Dual \eqref{eq:dualnoise}} It is illustrated in \cref{fig:geom2}, where $\s_1, \s_2$ are two chosen data-points. The blue shaded polytope is $K^\circ$. At $\lambda = \lambda_1$, the point $\lambda_1 \x$ lies in the interior of $K^\circ$. Hence, $A_\lambda(\x)$ is empty and ${\rm supp}(\c^*(\x))$ is also empty. As $\lambda$ increases, a non-empty support is obtained for the first time at $\lambda = 1 / \gamma_{K^{\circ}}(\x)$. For all $\lambda_2 \x$ in the red shaded polyhedron, the projection $\d^*_{\lambda_2}(\x) = {\rm Proj}_{K^\circ}(\lambda_2 \x)$ lies on the face $F$. As $\lambda$ increases further we reach the green polyhedron. Further increases in $\lambda$ do not change the dual solution, which will always remain at the vertex $\d^*_{\lambda_3}(\x)$.%This can be seen algebraically as setting $\lambda \to \infty$ in \eqref{eq:dualnoise} is equivalent to
%Problem \eqref{eq:dualnoise} the same as
%taking $\epsilon = 0$.
%where again $s_i$ is a sign variable.\rene{This does not define $s_i$. Please write $s_i = sign(\langle x_i , d^*\rangle)$ AP: Fixed.}
% Note that \eqref{eq:activelambda} is a much simpler definition than \eqref{eq:active} as the set of optimal dual solutions now has a single element.
Geometrically, $A_\lambda(\x)$ identifies the face of $K^\circ$ which contains the projection of $\lambda \x$, if $A_\lambda(\x)$ is non-empty (otherwise, $\lambda \x$ lies inside the polyhedron $K^\circ$). The support of the primal solution, $\c^*(\x)$, is a subset of $A_\lambda$, \ie~ ${\rm supp}(\c^*(\x)) \subseteq A_\lambda(\x)$. %This geometry is shown in \cref{fig:geom2}. %(see the caption for further details).
% explains how ${\rm Proj}_{K^\circ}(\lambda \x)$ and $A_\lambda(\x)$ vary as $\lambda$ increases from $0$ to $\infty$.%\paris{It might be better to use the same drawing style of Fig.1 i.e., draw $K$ as well. AP: Agreed, fixed.}.
Note that whenever two points, say $\x, \x'$, both lie in the same shaded polyhedron (red or green), their projections would lie on the same face of $K^\circ$. We now show this formally, in the main theorem of this section.
\begin{restatable}[]{theorem}{noisyl}
% \begin{theorem}
\label{lem:noisyl2}
The set of active constraints $A_\lambda$ defined in \eqref{eq:activelambda} is robust, \ie, $A_\lambda(\x') = A_\lambda(\x)$ for all $\lambda \x' \in C(\x)$, where $C(\x)$ is the polyhedron defined as
\begin{equation}
C(\x) = F(\x) + V(\x), \label{eq:conenoisyl2}
\end{equation}
with $F \subseteq K^\circ$ being a facet of the polyhedron $K^\circ$ that $\x$ projects to, defined as
\begin{equation}
F(\x) = \left\{ \d \ \left| \ \begin{aligned}&\bt_i^\top \d = 1, \forall \bt_i \in A_\lambda(\x)\\&\bt_i^\top \d < 1, {\rm otherwise}\end{aligned}\right. \right\}, \label{face}
\end{equation}
and $V$ being the cone generated by the constraints active at (i.e., normal to) $F$, defined as
\begin{equation}
V(\x) = \left\{ \sum_{\bt_i \in A_\lambda(\x)} \alpha_i \bt_i \colon \alpha_i \geq 0, \forall \bt_i \in A_\lambda(\x) \right\}. \label{cone} %\rene{$S$ or $A_\lambda$? AP: Fixed.}
\end{equation}
% \end{theorem}
\end{restatable}
The proof of \cref{lem:noisyl2} utilizes the geometry of the problem and properties of the projection operator, and is presented in \cref{app:noisyl2}. We can now use this result to construct a robust classifier:%, as follows:%
\begin{lemma}
Define the \emph{dual} classifier as
\begin{equation}
g_\lambda(\x) = \textsc{Aggregate}(\{y_i \colon \bt_i \in A_\lambda(\x)\}), \label{eq:g}
\end{equation}
where $\textsc{Aggregate}$ is any deterministic mapping from a set of labels to $\cY$, e.g., \textsc{Majority}. Then, for all $\x' \in C(\x)$ as defined in \cref{lem:noisyl2}, $g_\lambda$ is certified to be robust, i.e., $g_\lambda(\x') = g_\lambda(\x)$.
\end{lemma}
%
\myparagraph{Implications} Having obtained a certifiably robust classifier $g$, we pause to understand some implications of the theory developed so far. We observe that the certified regions in \cref{lem:noisyl2} are not spherical, \ie, the attacker can make additive perturbations having large $\ell_2$ norm but still be unable to change the label predicted by $g$ (see \cref{illustration}). This is in contrast to the $\ell_2$ bounded certified regions that can be obtained by most existing work on certification schemes, and is a result of modelling data structure while constructing robust classifiers. Importantly, however, note that we do not assume that the attack is restricted to the subspace. %\ambarwriting{Have the figure about conical certificates, and comparison to L2 certificates here.}
\myparagraph{Connections to Classical Results}
For $\epsilon = 0$, \cref{eq:primalnoiseraw} is known as the primal form of the Basis Pursuit problem, and has been studied under a variety of conditions on $\S$ in the sparse representation and subspace clustering literature \cite{foucart2013invitation,donoho2003optimally,soltanolkotabi2012geometric,you2015geometric,li2018geometric,Kaba:ICML21}. Given an optimal solution $\c^*(\x)$ of this basis pursuit problem, how can we accurately predict the label $y$? One ideal situation could be that all columns in the support predict the same label, \ie, $y_i$ is identical for all $i \in \text{supp}(\c^*(\x))$. Indeed, this ideal case is well studied, and is ensured by necessary \cite{Kaba:ICML21} and sufficient \cite{soltanolkotabi2012geometric,you2015geometric,li2018geometric} conditions on the geometry of the subspaces $S_1, \ldots, S_K$. Another situation could be that the \emph{majority} of the columns in the support predict the correct label. In this case, we could predict $\texttt{Majority}(\{y_i \colon i \in {\rm supp}(\c^*(\x))\})$ to ensure accurate prediction. \cref{lem:noisyl2} allows us to obtain robustness guarantees which work for \emph{any} such aggregation function which can determine a single label from the support. Hence, our results can guarantee robust prediction even when classical conditions are not satisfied. Lastly, note that our \cref{lem:noisyl2} shows that the entire active set remains unperturbed -- In light of the recent results in \cite{sulam2020adversarial}, this could be relaxed for specific choices of maps acting on the estimated support.
\section{Experiments} \label{sec:expt}
In this section, we will compare our certified defense derived in \cref{sec:subspace} to a popular defense technique called Randomized Smoothing (RS) \cite{cohen2019certified}, which can be used to obtain state-of-the-art certified robustness against $\ell_2$ perturbations. RS transforms any given classifier $f \colon \cX \to \cY$ to a certifiably robust classifier $g^{\rm RS}_\sigma \colon \cX \to \cY$ by taking a majority vote over inputs perturbed by Gaussian\footnote{In reality, the choice of the noise distribution is central to determining the type of certificate one can obtain \cite{pal2023understanding,yang2020randomized}, but Gaussian suffices for our purposes here.} noise $\mathcal{N}(0, \sigma^2 I)$, \ie,
\begin{equation}
g^{\rm RS}_\sigma(x) = {\rm Smooth}_\sigma(f) = \argmax_{k \in \cY} \Pr_{v \sim \mathcal{N}(0, \sigma^2 I)} (f(x + v) = k). \label{smoothing}
\end{equation}
Then, at any point $x$, $g^{\rm RS}_\sigma$ can be shown to be certifiably robust to $\ell_2$ perturbations of size at least $r^{\rm RS}(x) = \sigma \Phi^{-1}(p)$ where $p = \max_{k \in \cY} \Pr_{v \sim \mathcal{N}(0, \sigma^2 I)} (f(x + v) = k)$ denotes the maximum probability of any class under Gaussian noise.
% \paragraph{Challenges in Comparing Certificates}
% \begin{wrapfigure}{r}{0.4\textwidth}
\begin{figure}[h!]
% \vspace{-4ex}
\centering
\includegraphics[width=0.4\textwidth]{fig/expt_comparison}
\caption{Comparing polyhedral and spherical certificates. Details in text.}
\label{illustration}
\vspace{-4mm}
\end{figure}
% \end{wrapfigure}
It is not immediately obvious how to compare the certificates provided by our method described above and that of RS, since the sets of the space they certify are different. The certified region obtained by RS, $C^{\rm RS}(x) = \{\bar x \colon \|x - \bar x\|_2 \leq r(x)\}$, is a sphere (orange ball in \cref{illustration}). In contrast, our certificate $C_\lambda(x)$ from \cref{lem:noisyl2} is a polyhedron (resp., blue trapezoid), which, in general, is neither contained in $C^{\rm RS}(x)$, nor a superset of $C^{\rm RS}(x)$. Additionally, our certificate has no standard notion of \emph{size}, unlike other work on elliptical certificates \cite{eirasancer}, making a size comparison non-trivial. To overcome these difficulties, we will evaluate two notions of \emph{attack size}: in the first, we will compare the $\ell_2$ norms of successful attacks projected onto our polyhedron, % (here we expect $C_\lambda(x)$ to be ``better'' than $C^{\rm RS}(x)$),
and in the second, we will compare the minimum $\ell_2$ norm required for a successful attack. %(here we expect $C_\lambda(x)$ to be ``worse'' than $C^{\rm RS}(x)$).
We will then combine our method with RS to get the best of both worlds, \ie, the green shape in \cref{illustration}. In the following, we present both these evaluations on the MNIST \cite{lecun} dataset, with each image normalized to unit $\ell_2$ norm.
\myparagraph{Comparison along Projection on $C_\lambda(x)$} For the first case, we investigate the question: \emph{Are there perturbations for which our method is certifiably correct, but Randomized Smoothing fails?} For an input point $x$, we can answer this question in the affirmative by obtaining an adversarial example $\bar x$ for $g^{\rm RS}$ such that $\bar x$ lies inside our certified set $C_\lambda(x)$. Then, this perturbation $v = x - \bar x$ is certified by our method, but has $\ell_2$ norm larger than $r^{\rm RS}(x)$ (by definition of the RS certificate).
To obtain such adversarial examples, we first
% examples $\bar x \in C_\lambda(x)$ such that the $\ell_2$ norm of $v = x - \bar x$ is greater than the certified radius given by RS, \ie, $\|v\|_2 > r^{\rm RS}(x)$. First, we
train a standard CNN classifier $f$ for MNIST, and then use RS\footnote{\label{note1} Further details (e.g., smoothing parameters, certified accuracy computation) are provided in \cref{app:expt}.} to obtain the classifier $g^{\rm RS}_\sigma$. Then, for any $(x, y)$, we perform projected gradient descent to
% to find an adversarially perturbed point $\bar x$ lying in our certified region $C_\lambda(x)$, such that $\|\x - \bar \x\|_2 \leq \epsilon$.
% More specifically, we
obtain $\bar x = x^T$ by performing the following steps $T$ times,
starting with $x^0 \gets x$:
\begin{gather}
\rom{1.}\ x^t \gets {\rm Proj}_{B_{\ell_2}(x, \epsilon)}\Big(x^{t-1} + \eta \nabla_x {\rm Loss}(g^{\rm RS}_\sigma(x^t), y)\Big) \qquad \rom{2.}\ x^t \gets {\rm Proj}_{C_\lambda(x)} (x^t) \label{pgdmod}
\end{gather}
% \begin{gather*}
% \rom{1.}\ u^t \gets x^{t-1} + \eta \nabla_x g^{\rm RS}(x^t) \quad \rom{2.}\ v^t \gets u^t - x \quad
% \rom{3.}\ v^t \gets (\epsilon / \|v^t\|_2) v^t \\ \rom{4.}\ x^t \gets {\rm Proj}_{C_\lambda(x)} (x + v^t).
% \end{gather*}
% (1) $u^t \gets x^{t-1} + \eta \nabla_x g^{\rm RS}(x^t)$, (2) $v^t \gets u^t - x$, (3) $v^t \gets (\epsilon / \|v^t\|_2) v^t$, and (4) $x^t \gets {\rm Proj}_{C_\lambda(x)} (x + v^t)$.
Unlike the standard PGD attack (step \rom{1}), the additional projection (step \rom{2}) is not straightforward, and requires us to solve a quadratic optimization problem, which can be found in \cref{app:expt}. %\ambarwriting{Look at all mentions of Appendix, and write which section in the Appendix exactly}%
We can now evaluate $g^{\rm RS}_\sigma$ on these perturbations to empirically estimate the robust accuracy over $C_\lambda$, \ie,
% With these perturbations, we plot an upper-bound on the robust accuracy of $g^{\rm RS}$ over our certified set $C_\lambda(x)$. More formally, we plot an empirical estimate of the quantity
\begin{equation*}
{\rm ProjectionRobustAcc}(\epsilon) = \Pr_{x, y \sim p_{\rm MNIST}} \Big( \exists \bar x \in B_{\ell_2}(x, \epsilon) \cap C_\lambda(x) \text{ such that } g^{\rm RS}(\bar x) \neq y \Big).
\end{equation*}
\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-4ex}
\centering
\includegraphics[width=0.4\textwidth]{fig/comp_g_expanded_recolor.pdf}
\vspace{-5ex}
\caption{Comparing RS with Our method for adversarial perturbations computed by repeating Steps I, II \eqref{pgdmod}.}
\label{fig:comparison1}
\vspace{-5ex}
\end{wrapfigure}
The results are plotted in \cref{fig:comparison1}, as the dotted curves. We also plot the certified accuracies\cref{note1} for comparison, as the solid curves. We see that the accuracy certified by RS drops below random chance (0.1) around $\epsilon = 0.06$ (solid red curve). Similar to other certified defenses, RS certifies only a subset of the true robust accuracy of a classifier in general. This true robust accuracy curve is pointwise upper-bounded by the empirical robust accuracy curve corresponding to any attack, obtained via the steps $\rom{1},\rom{2}$ described earlier (dotted red curve). We then see that even the upper-bound drops below random chance around $\epsilon = 0.4$, suggesting that this might be a large enough attack strength so that an adversary only constrained in $\ell_2$ norm is able to fool a general classifier. However, we are evaluating attacks lying on our certified set and it is still possible to recover the true class (blue solid curve), albeit by our specialized classifier $g_\lambda$ suited to the data structure. Additionally, this suggests that our certified set contains useful class-specific information -- this is indeed true, and we present some qualitative examples of images in our certified set in \cref{app:expt}. \begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-10pt}
\centering
%\includegraphics[width=0.4\textwidth]{fig/ours+RS_recolor.pdf}
\includegraphics[width=0.4\textwidth]{fig/ours+RS_recolor_500.pdf}
\vspace{-20pt}
\caption{Comparing Certified Accuracy after combining our method with RS.}
\label{fig:comparison2}
% \vspace{-40pt}
\vspace{-15pt}
\end{wrapfigure} To summarize, we have numerically demonstrated that \emph{exploiting data structure in classifier design leads to certified regions capturing class-relevant regions beyond $\ell_p$-balls}.
\myparagraph{Comparison along $\ell_2$ balls} For the second case, we ask the question: \emph{Are there perturbations for which RS is certifiably correct, but our method is not?} When an input point $x$ has a large enough RS certificate $r^{\rm RS}(x) \geq r_0$, some part of the sphere $B_{\ell_2}(x, r^{\rm RS}(x))$ might lie outside our polyhedral certificate $C_\lambda(x)$ (blue region in \cref{illustration}). In theory, the minimum $r_0$ required can be computed via an expensive optimization program that we specify in \cref{app:expt}. In practice, however, we use a black-box attack \cite{chen2020hopskipjumpattack} to find such perturbations. We provide qualitative examples and additional experiments on CIFAR-10 in \cref{app:expt}. %\ambarwriting{Link to appendix once these examples are there}%
\myparagraph{Combining Our Method with RS}
We now improve our certified regions using randomized smoothing. For this purpose, we treat our classifier $g_\lambda$ as the base classifier $f$ in \eqref{smoothing}, to obtain ${\rm Smooth_\sigma(g_\lambda)}$ (abbreviated as ${\rm Ours}^{\rm RS}_\sigma$). We then plot the $\ell_2$ certified accuracy\cref{note1} \cite[Sec 3.2.2]{cohen2019certified} in \cref{fig:comparison2}, where note that, as opposed to \cref{fig:comparison1}, the attacks are \emph{not} constrained to lie on our certificate anymore. %
%We observe that not only does randomized smoothing enable us to obtain a $\ell_2$ robustness certificate, we can in fact slightly improve (note the $10\times$ zoom on the $x$-axis compared to \cref{fig:comparison1}) over the certificates obtained by RS on its own in some regimes (green curve vs. red curve in \cref{fig:comparison2}).
As a final remark, we note that our objective in \cref{fig:comparison2} was simply to explore RS as a method for obtaining an $\ell_2$ certificate for our method, and we did not tune our method or RS for performance. In particular, we believe that a wide array of tricks developed in the literature for improving RS performance \cite{pal2023understanding,smoothadv,yang2020randomized} could be employed to improve the curves in \cref{fig:comparison2}. We now discuss our work in the context of existing literature in adversarial robustness, and sparse representation learning. %
%
% In this section, we empirically compare our certificates with those that exist in the literature. We will be working with the MNIST dataset \ambarwriting{NeurIPS guidelines require citing MNIST authors}, with all images normalized to unit $\ell_2$ norm. Our aim will be to demonstrate that the defense studied in \cref{sec:subspace} can provably certify adversarial perturbations which have an $\ell_2$ norm too high to be certified by existing approaches.
%
%For any given data-point $\x$, recall that our certificate is a data-dependent polyhedron $C_\lambda(\x)$. Firstly, in \cref{fig:reb3}, we qualitatively demonstrate that this polyhedron captures valid perturbations which might have large $\ell_2$ norms.
%
%Next, we train a standard classifier for MNIST (details of the training process can be found in the Appendix). We then generate adversarial attacks for this classifier using the Projected Gradient Descent (PGD) attack. In the standard PGD attack, the projection step involves projection onto an $\ell_2$ ball. In our case, we have an additional projection onto
%our certificate $C(\x)$. In other words, we perform the following steps for $t = 1 \ldots T$ times \ambarwriting{Mention the optimization problem for certify in QP form, it might not be trivial to the reader.}, starting with $\x^0 \gets \x$, for each each image $\x$ to produce the attacked image $\x + \v^T$: (1) $\u^t \gets \x^{t-1} + \eta \nabla_\x f(\x^t)$, (2) $\v^t \gets \u^t - \x$, (3) $\v^t \gets (\epsilon / \|\v^t\|_2) \v^t$, and (4) $\x^t \gets {\rm Proj}_{C(\x)} (\x + \v^t)$.
%
%The above involves a projection to the $\epsilon$ ball $B(\x, \epsilon)$ in step (3) to ensure that the perturbed image remains close to the original image. Then, we project to the certified region $C(\x)$ to ensure that our defense classifies the point correctly. Due to the semantically meaningful nature of our certificate, this additional projection ensures that the identity of the image does not change after perturbation, which is likely once the perturbation becomes large enough. We take $T = 20, \eta = 0.5$ and vary $\epsilon$ to generate attacks of different strengths. \cref{fig:reb_qual} then shows some examples where our adversarially trained classifier $f$ misclassifies the sample, i.e., $\x + \v^T$ is an adversarial image with perturbation size $\epsilon$, but our method classifies it correctly. \cref{fig:reb_qual} shows that our method is able to classify perturbations like $(b)$ correctly, where the perturbed image has the same class as the original, but the perturbation has a large $\ell_2$ norm.
%
% \begin{figure}[ht!]
% \centering
% \includegraphics[width=\textwidth]{fig/rebuttal_qual}
% \caption{For each pair of images, the first image shows the benign input $\x$, the second image the attacked input projected to the certified region, i.e., ${\rm Proj}_{C(\x)}(\x + \v)$, and the number shows the norm $\|\v\|_2$. The title shows the label output by a standard CNN trained on MNIST. In each pair, it can be seen that $\x$ is classified correctly, but an unrestricted perturbation causes ${\rm Proj}_{C(\x)}(\x + \v)$ to be misclassified. However, since ${\rm Proj}_{C(\x)}(\x + \v)$ lies inside the certified set, our method classifies each of the images correctly after perturbation. Note that all images $\x$ are normalized to have $\ell_2$ norm $1$, and all images are pixel-wise clipped to $[0, 1]$ before the classification $f(\x)$.}
% \label{fig:reb_qual}
% \end{figure}
%\section{Discussion and Conclusion}
\section{Discussion and Related Work}
\label{sec:relwork}
% \ambar{Include References to Ben thesis here.}
%For our concentration results,
Our proof techniques utilize tools from high-dimensional probability, and have the same flavor as recent impossibility results for adversarial robustness \cite{dohmatob,goldstein,madryB}. Our geometric treatment of the dual optimization problem %\ambarwriting{Since the paper now heavily talks about data concentration, it makes sense to have a greater emphasis on those works here}
is similar to the literature on sparse-representation \cite{donoho2003optimally,foucart2013invitation} and subspace clustering \cite{soltanolkotabi2012geometric,soltanolkotabi2014robust,you2015geometric,Kaba:ICML21}, which is concerned with the question of representing a point $\x$ by the linear combination of columns of a dictionary $\S$ using sparse coefficients $\c$. As mentioned in \cref{sec:subspace}, there exist geometric conditions on $\S$ such that all such candidate vectors $\c$ are \emph{subspace-preserving}, i.e., for all the indices $i$ in the support of $\c$, it can be guaranteed that $\s_i$ belongs to the correct subspace. On the other hand, the question of classification of a point $\x$ in a union of subspaces given by the columns of $\S$, or subspace classification, has also been studied extensively in classical sparse-representation literature \cite{wright2008robust,wright2010sparse,cappelli2002subspace,laaksonen1997subspace,yu2015semi,yu2015semi,yu2015semi,naseem2010linear,hui2012sparse,li2013nearest}. The predominant approach is to solve an $\ell_1$ minimization problem to obtain coefficients $\c$ so that $\x = \S \c + \e$, and then predict the subspace that minimizes the representation error. Various \emph{global} conditions can be imposed on $\S$ to guarantee the success of such an approach \cite{wright2008robust}, and its generalizations \cite{elhamifar2011robust,elhamifar2012block}. Our work differs from these approaches in that we aim to obtain conditions on perturbations to $\x$ that ensure accurate classification, and as such we base our robust classification decision upon properties of solutions of the dual of the $\ell_1$ problem.
Our results are complementary to \cite{pydi2020adversarial,bhagoji2019lower}, who % and are derived in a more general multi-class setting. \cite{pydi2020adversarial,bhagoji2019lower}
obtain a %nice
lower bound on the robust risk $R(f, \epsilon)$, in a binary classification setting, in terms of the Wasserstein distance $D$ between the class conditionals $q_0$ and $q_1$, \ie, $R(f, \epsilon) \geq 1 - D(q_0, q_1)$. Using this result, \cite{pydi2020adversarial,bhagoji2019lower} roughly state that the robust risk increases as the class conditionals get closer, \ie, it becomes more likely to sample $x \sim q_0, x' \sim q_1$, such that $\|x - x'\| \leq \epsilon$. In comparison, for two classes, our \cref{th:strongconc} roughly states that for low robust risk, $q_0$ and $q_1$ should be concentrated on small-volume subsets separated from one another. Thus, the message is similar, while our \cref{th:strongconc} is derived in a more general multi-class setting. Note that \cite{pydi2020adversarial,bhagoji2019lower} do not explicitly require small-volume subsets, but this is implicit, as $1 - D(q_0, q_1)$ is large due to concentration of measure when $q_0, q_1$ do not concentrate on very small subsets. We provide an analogue of the empirical results of \cite{pydi2020adversarial,bhagoji2019lower} in \cref{app:cifar10lbcomparison}.
Our notion of concentration is also related to concentration of a measure $\mu$, as considered in \cite{mahloujifar2019curse,mahloujifar2019empirically,zhang2021understanding}, which denotes how much the $\mu$-measure of any set \emph{blows up} after expansion by $\epsilon$. %(i.e., the minimum $\mu(A^{+\epsilon})$ over all sets $A$ with $\mu(A) \geq 0.5$).
Under this definition, the uniform measure has a high degree of concentration in high dimensions, and this is called the concentration of measure phenomenon. In contrast, our \cref{def:conc} of concentrated data distributions can be seen as a \emph{relative} notion of concentration with respect to the uniform measure $\mu$, in that we call a class conditional $q$ concentrated when it assigns a large $q$-measure to a set of very small volume, (\ie, $q(S)$ is high whereas $\mu(S)$ is very low). In essence, \cref{def:conc} defines concentration relative to the uniform distribution, whereas \cite{mahloujifar2019curse,mahloujifar2019empirically,zhang2021understanding} define concentration independently. \cref{def:conc} is useful in the context of adversarial robustness as it allows us to separate the concentration of data distributions (which is unknown) from the concentration of the uniform measure in high dimensions (for which there is a good understanding). This allows us to derive results in the non-realizable setting, where errors are measured against a probabilistic ground truth label $Y$, which is strictly more general than the realizable setting which requires a deterministic ground truth classifier $f^*$. As a result of this realizable setting, the analysis in \cite{gilmer2018relationship,mahloujifar2019curse,mahloujifar2019empirically,zhang2021understanding} needs to assume a non-empty error region $A$ for the learnt classifier $g$ with respect to $f^*$, in order to reason about $\epsilon$-expansions $A^{+\epsilon}$. The results in \cite{mahloujifar2019curse,mahloujifar2019empirically} indicate that the robust risk grows quickly as the volume of $A$ increases. However, humans seem to be a case where the natural accuracy is not perfect (e.g., we might be confused between a $6$ and a poorly written $5$ in MNIST), yet we seem to be very robust against small $\ell_2$ perturbations. This points to a slack in the analysis in \cite{mahloujifar2019curse,mahloujifar2019empirically}, and our work fills this gap by considering $\epsilon$ expansions of a different family of sets.
Finally, our work is also related to recent empirical work obtaining robust classifiers by \emph{denoising} a given input $\x$ of any adversarial corruptions, before passing it to a classifier \cite{samangouei2018defense,moosavi2018divide}. However, such approaches lack theoretical guarantees, and might be broken using specialized attacks \cite{niu2020limitations}. Similarly, work on improving
the robustness of deep network-based classifiers by adversarial training off the data-manifold can be seen as an empirical generalization of our attack model \cite{jin2020manifold,moosavi2019robustness,zhang2021manifold,lin2020dual}. More generally, it has been studied how adversarial examples relate to the underlying data-manifold \cite{stutz2019disentangling,khoury2018geometry,ma2018characterizing}.
Recent work also studies the robustness of classification using projections onto a single low-dimensional linear subspace \cite{awasthi2020adversarial,awasthi2021adversarially,somayeh}. %Such works are close to us in spirit, as they use projections onto a single low-dimensional linear subspace to provide robustness certificates.
The work in \cite{awasthi2020adversarial} studies an the attack model of bounded $\ell_2, \ell_\infty$ attacks, and they provide robustness certificates by obtaining guarantees on the distortion of a data-point $\x$ as it is projected onto a single linear subspace using a projection matrix $\Pi$. In contrast, our work can be seen as projecting a perturbed point onto a union of multiple low-dimensional subspaces. The resultant richer geometry allows us to obtain more general certificates.% that are unbounded by any $\ell_p$ norm.
%
%\rene{Connecting within subspace and out of subspace perturbations to within manifold and out-of manifold work would be appropriate}
\section{Conclusion and Future Work}
\label{sec:conclusion}
To conclude, we studied conditions under which a robust classifier exists for a given classification task. We showed that concentration of the data distribution on small subsets of the input space is necessary for any classifier to be robust to small adversarial perturbations, and that a stronger notion of concentration is sufficient. We then studied a special concentrated data distribution, that of data distributed near low-dimensional linear subspaces. For this special case of our results, we constructed a provably robust classifier, and then experimentally evaluated its benefits w.r.t. known techniques. %this paper, we studied the question of adversarial robustness for a classification task where the training data lies on a union of low-dimensional linear subspaces. In this setting, we constructed robust classifiers for the unbounded, on-subspace and out-of-subspace attack models, and obtained their associated robustness guarantees. We then applied the resultant theory
%developed
% to classical $\ell_2$ bounded attacks and gave optimization algorithms for computing the certified radii.
%One of the limitations of our work is that
For the above special case, we assume access to a \emph{clean} dataset $\S$ lying perfectly on the union of low-dimensional linear subspaces, while in reality one might only have access to noisy samples. In light of existing results on noisy subspace clustering \cite{wang2013noisy}, an immediate future direction is to adapt our guarantees to support noise in the training data. Similarly, while the assumption of low-dimensional subspace structure in $\S$ enables us to obtain novel unbounded robustness certificates, real world datasets might not satisfy this structure. We hope to mitigate this limitation by extending our formulation to handle data lying on a general image manifold in the future.
%\section{Practical Applicability of the Subspace-Threat Models}
%In this section, we will provide some qualitative examples of adversarial attacks covered by our threat models, and demonstrate their applicability on MNIST. Even though MNIST is considered a ``toy'' dataset compared to modern classification datasets in computer vision, the task of obtaining provable guarantees for MNIST is far from being solved. Consider Jeong et. al.\cite{jeong2020consistency}, who obtain the best provable guarantees on MNIST to the best of our knowledge. At a perturbation of size $1.5$ in the $\ell_2$ norm, they are able to certify an accuracy of $82.2\%$, and this number drops to $28.0\%$ at a perturbation size $2$. For comparison, a few images with $\ell_2$ perturbations of size $2$ and $1.5$ are shown in Figure \cref{fig:comparison1}. It should be clear that as humans we are able to quite easily predict the correct digit at these perturbation levels, while certified methods barely work.
%{\color{blue} In our paper, all the inputs are normalized to $[0, 1]$, and the $\epsilon$ values cannot be directly compared.}
%
%The situation is even more dire when we consider semantic perturbations, which do not necessarily have a small $\ell_p$ norm.
\begin{ack}
We would like to thank Amitabh Basu for helpful insights into the optimization formulation for the largest $\ell_2$ ball contained in a polyhedron. This work was supported by DARPA grant HR00112020010, NSF grants 1934979 and 2212457, and the NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning (NSF grant 2031985, Simons grant 814201).
\end{ack}
\bibliographystyle{plain}
\bibliography{refs}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\section{Proof of \cref{th:concentration}} \label{proof:concentration}
We will make the technical assumption in \cref{sec:necessarycond} precise. Recall that all quantities are normalized so that $\cX$ is an $\ell_2$ ball of radius $1$, i.e., $\cX = B_{\ell_2}(0, 1)$. Recall that $q_k \colon \cX \to \R_{\geq 0}$ denotes the conditional distribution for class $k \in \cY$, having support $Q_k$. %We will impose a technical restriction %\ambarwriting{I wonder whether this should be fleshed out completely here, or moved to the appendix. A casual reader might take this $\epsilon \leq 0.1$ to be evidence that our results are only true for low $\epsilon$.}
%to ensure that an adversarial perturbation on any data-point sampled from $p$ does not take the resultant attacked point outside the domain $\cX$.
We will assume that there is a sufficient gap between the supports and the boundary of the domain, i.e., $Q_k \subseteq B_{\ell_2}(0, c)$, and that the $\ell_2$-adversarial attack has strength $\epsilon \leq c$, for some constant $c$, say $c = 0.1$. %The numerical values are arbitrary here, and taken to simplify notation.
%\ambartechnical{Provide the more complicated proof without this assumption, and state how the concentration criterion changes to a much less interpretable version}
\concentration*
\begin{proof}
Note that %\ambartechnical{Note that we don't need $\cX$ to be the full dimensional ball, we can do with a low-dimensional $\cX$ provided the adversarial examples, and the volumes are also low-dimensional. If this synergy does not exist, Th 1 can be \emph{false}: have one plane as class 0, other far apart plane as class 1, and a robust classifier exists for full-dimensional $\ell_2$ bounded attacks even when nothing is concentrated in the lower-dimensional volume}
the definition \eqref{robustness-defn} of $(\epsilon, \delta)$-robustness may be rewritten as
\begin{align}
R(f, \epsilon)
&= \sum_k p_{X|Y}(X \text{admits an $\epsilon$-adversarial example} | Y = k) p_Y(y = k) \nonumber \\
&= \sum_k q_k(\{x \in \cX \colon \exists \bar x \in B(x, \epsilon) \text{ such that } f(\bar x) \neq k \}) p_Y(y = k), \label{implicit}
\end{align}
where $q(S)$ denotes the $q$-measure of the set $S$.
We are given $f$ such that $R(f, \epsilon) \leq \delta$, and we want to find a set $S \subseteq \cX$ over which $p$ concentrates.
\begin{gather}
R(f, \epsilon) \leq \delta \nonumber \\
\implies \sum_k q_k(\{x \in \cX \colon \exists \bar x \in B(x, \epsilon) \cap \cX \text{ such that } f(\bar x) \neq k\}) p_Y(y = k) \leq \delta \label{cvxcomb}
\end{gather}
Since $\sum_k p_Y(y = k) = 1$, the LHS in \eqref{cvxcomb} is a convex combination, and we have
\begin{gather}
\implies \exists \hat k \ q_{\bar k}(\{x \in \cX \colon \exists \bar x \in B(x, \epsilon) \cap \cX \text{ such that } f(\bar x) \neq \hat k\}) \leq \delta, \label{advmeasure} \\
\implies q_{\bar k}(U) \leq \delta, \text{ where } \nonumber \\
U = \{x \in \cX \colon \exists \bar x \in B(x, \epsilon) \cap \cX \text{ such that } f(\bar x) \neq \hat k\}. \nonumber
\end{gather}
%to be set being measured in \eqref{advmeasure}.
We will express $q_{\hat k}(U)$ in terms of the classification regions of $f$.
Let $A \subseteq \cX$ denote the region where the classifier predicts $\hat k$, i.e. $A = \{x \in \cX \colon f(x) = \hat k\}$. Define $A^{-\epsilon}$ to be the set of all points in $A$ at a $\ell_2$ distance atleast $\epsilon$ from the boundary, i.e. $A^{-\epsilon} = \{x \in A \colon B(x, \epsilon) \subseteq A\}$. Now consider any point $x \in \cX$. We have the following 4 mutually exclusive, and exhaustive cases:
\begin{enumerate}
\item $x$ lies outside $A$, i.e., $x \not \in A$. In this case, $x \in U$ trivially as $f(x) \neq k$.
\item $x \in A$, and the entire $\epsilon$-ball around $x$ lies inside $A$, i.e., $B(x, \epsilon) \subseteq A$. In this case, every point in the $\epsilon$-ball around $x$ is classified into class $\hat k$, and $x$ has no $\epsilon$-adversarial example for class $\hat k$. Note that the set of all these points is $A^{-\epsilon}$ by definition.
\item $x \in A$, and some portion of the $\epsilon$-ball around $x$ lies outside $A$, i.e., $B(x, \epsilon) \not \subseteq A$, and, there is a point $\bar x \in B(x, \epsilon) \cap \cX$ such that $f(\bar x) \neq k$. In this case, $\bar x$ is an adversarial example for $x$, and $\bar x \in U$.
\item $x \in A$, and some portion of the $\epsilon$-ball around $x$ lies outside $A$, i.e., $B(x, \epsilon) \not \subseteq A$, \emph{but there is no point $\bar x \in B(x, \epsilon) \cap \cX$ such that $f(\bar x) \neq k$}. This can happen only when for all $\bar x \in B(x, \epsilon) \setminus A$ we have $\bar x \not \in \cX$. In other words, any $\epsilon$ perturbation that takes $x$ outside $A$, also takes it outside the domain $\cX$. This implies that $B(x, \epsilon) \not \subseteq \cX$, which in turn means that the distance of $x$ from the boundary of $\cX$ is atmost $\epsilon$.
\end{enumerate}
We see that $U$ comprises of the points covered in cases (1) and (3). Recall that we assumed that any point having distance less than $0.1$ to the boundary of $\cX$ does not lie in the support of $q_{\bar k}$. But all points covered in case (4) lie less than $\epsilon$-close to the boundary, and $\epsilon \leq 0.1$. This implies that $q_{\bar k}$ assigns $0$ measure to the points covered in case (4). Together, cases (1), (3) and (4) comprise the set $(\cX \setminus A) \cup (A \setminus A^{-\epsilon})$. Thus, we have shown
\begin{gather}
q_{\hat k}(U) = q_{\hat k}(\cX \setminus A^{-\epsilon}) \nonumber \\
\implies q_{\hat k}(A^{-\epsilon}) \geq 1 - \delta. \label{lbdelta}
\end{gather}
Next, we will show that the set $A^{-\epsilon}$ is exponentially small, by appealing to the Brunn-Minkowski inequality, which states that for compact sets $E, F \subset \R^n$,
\begin{equation*}
{\rm Vol}(E + F)^{\frac{1}{n}} \geq {\rm Vol}(E)^{\frac{1}{n}} + {\rm Vol}(F)^{\frac{1}{n}},
\end{equation*}
where ${\rm Vol}$ is the $n$-dimensional volume in $\R^n$, and $E + F$ denotes the minkowski sum of the sets $E$ and $F$. Applying the inequality with $E = A^{-\epsilon}$ and $F = B_{\ell_2}(0, \epsilon)$, we have
\begin{align}
{\rm Vol}(A^{-\epsilon} + B_{\ell_2}(0, \epsilon))^{\frac{1}{n}} &\geq {\rm Vol}(A^{-\epsilon})^{\frac{1}{n}} + {\rm Vol}(B_{\ell_2}(0, \epsilon))^{\frac{1}{n}} \nonumber \\
\implies {\rm Vol}(A)^{\frac{1}{n}} &\geq {\rm Vol}(A^{-\epsilon})^{\frac{1}{n}} + \epsilon {\rm Vol}(B_{\ell_2}(0, 1))^{\frac{1}{n}} \nonumber \\
\implies {\rm Vol}(A)^{\frac{1}{n}} &\geq {\rm Vol}(A^{-\epsilon})^{\frac{1}{n}} + \epsilon {\rm Vol}(A)^{\frac{1}{n}} \nonumber \\
\implies \frac{{\rm Vol}(A^{-\epsilon})^{\frac{1}{n}}}{{\rm Vol}(A)^{\frac{1}{n}}} &\leq (1 - \epsilon) \nonumber \\
\implies {\rm Vol}(A^{-\epsilon}) &\leq {\rm Vol}(A)(1 - \epsilon)^n \leq {\rm Vol}(A) \exp (-\epsilon n) \nonumber \\
\implies {\rm Vol}(A^{-\epsilon}) &\leq \bar C \exp (-n \epsilon), \label{ubeps}
%\implies {\rm Vol}(A \setminus A^{-\epsilon}) &\geq {\rm Vol}(A)(1 - (1 - \epsilon)^n) \geq {\rm Vol}(A)(1 - \exp (-\epsilon n))
\end{align}
where $\bar C = {\rm Vol}(A)$. %is the volume of the unit ball in $n$ dimensions, and $c_2 = n$.
From \eqref{lbdelta} and \eqref{ubeps}, we see that $q_{\bar k}$ is $(\bar C, \epsilon, \delta)$-concentrated. Additionally, if the classes are balanced, we can use the fact that every term of \eqref{cvxcomb} can be atmost $K\delta$, and apply the above reasoning for each class to conclude that each $q_k$ is $(C_{\max}, \epsilon, K \delta)$-concentrated, for some suitable choice of $C_{\max}$, e.g., $C_{\max} = \max_k {\rm Vol} \{x \in \cX \colon f(x) = k\}$.
\end{proof}
\section{Proof of \cref{th:strongconc}} \label{proof:strongconc}
\strongconc*
\begin{proof}
For each $k \in \{1, 2, \ldots, K\}$, let $S_k$ be the support of the conditional density $q_k$. Recall that $S^{+\epsilon}$ is defined to be the $\epsilon$-expansion of the set $S$. Define $C_k$ to be the $\epsilon$-expanded version of the concentrated region $S_k$ but removing the $\epsilon$-expanded version of all other regions $S_{k'}$, as
\begin{equation*}
C_k = \left( S_k^{+\epsilon} \setminus \cup_{k' \neq k} S^{+\epsilon}_{k'} \right) \cap \cX.
\end{equation*}
We will use these regions to define our classifier $f \colon \cX \to \{1, 2, \ldots, K\}$ as
\begin{equation*}
f(x) = \begin{cases}
1, &\text{ if } x \in C_1 \\
2, &\text{ if } x \in C_2 \\
\vdots \\
K, &\text{ if } x \in C_K \\
1, &\text{ otherwise }
\end{cases}.
\end{equation*}
We will show that $R(f, \epsilon) \leq \delta + \gamma$, which can be recalled to be %\ambartechnical{All balls are open here, to prevent problematic cases of $\bar x$ exactly on the boundary of a classification region}
\begin{equation*}
R(f, \epsilon) = \sum_k q_k(\{x \in \cX \colon \exists \bar x \in B(x, \epsilon) \cap \cX \text{ such that } f(\bar x) \neq k\}) p_Y(y = k).
\end{equation*}
In the above expression, the $q_k$ mass is over the set of all points $x \in \cX$ that admit an $\epsilon$-adversarial example for the class $k$, as
\begin{equation}
U_k = \{x \in \cX \colon \exists \bar x \in B(x, \epsilon) \cap \cX \text{ such that } f(\bar x) \neq k\}. \label{unsafek}
\end{equation}
Define $C^{-\epsilon}_k$ to be the set of points in $C_k$ at a distance atleast $\epsilon$ from the boundary of $C_k$ as $C^{-\epsilon}_k = \{x \in C_k \colon B(x, \epsilon) \subseteq C_k\}$. For any point $x \in U_k$, we can find $\bar x \in B(x, \epsilon)$ from \eqref{unsafek} such that $\bar x \not \in C_k$, showing that $x \not \in C_k^{-\epsilon}$. Thus, $U_k$ is a subset of the complement of $C_k^{-\epsilon}$, i.e., $U_k \subseteq \cX \setminus C_k^{-\epsilon}$, and we have
\begin{equation*}
R(f, \epsilon) = \sum_k q_k(U_k) p_Y(y = k) \leq \sum_k (1 - q_k(C_k^{-\epsilon})) p_Y(y = k).
\end{equation*}
Now we will need to show a few properties of the $\epsilon$-contraction. Firstly, for a set $M = N \cap O$, we have $M^{-\epsilon} = N^{-\epsilon} \cap O^{-\epsilon}$, which can be seen as
\begin{align*}
M^{-\epsilon} &= \{x \colon x \in M, B(x, \epsilon) \subseteq M\} \\
&= \{x \colon x \in N, x \in O, B(x, \epsilon) \subseteq N, B(x, \epsilon) \subseteq O\} = N^{-\epsilon} \cap O^{-\epsilon}.
\end{align*}
Secondly, for a set $M = N^c$, where $c$ denotes complement, we have $M^{-\epsilon} = (N^{+\epsilon})^c$. This can be seen as
\begin{align*}
M^{-\epsilon} &= \{x \colon x \in M, B(x, \epsilon) \subseteq M\} = \{x \colon x \not \in N, B(x, \epsilon) \subseteq N^c\} \\
&= \{x \colon x \not \in N, \forall x' \in B(x, \epsilon) \ x' \not \in N\} \\
&= \{x \colon \forall x' \in B(x, \epsilon) \ x' \not \in N\} \\
\implies (M^{-\epsilon})^c &= \{x \colon \exists x' \in B(x, \epsilon) \ x' \in N\} \\
&= N^{+\epsilon}.
\end{align*}
Thirdly, for a set $M = N \setminus O$, we have $M^{-\epsilon} = (N \cap O^c)^{-\epsilon} = N^{-\epsilon} \cap (O^c)^{-\epsilon}$ by the first property, and then $N^{-\epsilon} \cap (O^c)^{-\epsilon} = N^{-\epsilon} \cap (O^{+\epsilon})^c$ by the second property. This implies
\begin{align*}
(N \setminus O)^{-\epsilon} = N^{-\epsilon} \setminus O^{+\epsilon}.
\end{align*}
Fourthly, for a set $M = N \cup O$, we have $M^c = N^c \cap O^c$. Taking $\epsilon$-contractions, and applying the first and second properties, we get $M^{+\epsilon} = N^{+\epsilon} \cup O^{+\epsilon}$.
Applying the above properties to $C_k^{-\epsilon}$, we have
\begin{align*}
C_k^{-\epsilon}
&= \left( S_k^{+\epsilon} \setminus \cup_{k' \neq k} S^{+\epsilon}_{k'} \right)^{-\epsilon} \cap \cX^{-\epsilon} \\
&= \left( S_k \setminus \big(\cup_{k' \neq k} S^{+\epsilon}_{k'} \big)^{+\epsilon} \right) \cap \cX^{-\epsilon} \\
&\supseteq \left( S_k \setminus \cup_{k' \neq k} S^{+2\epsilon}_{k'} \right) \cap \cX^{-\epsilon}.
\end{align*}
%Let $M = \cup_{k' \neq k} S^{+\epsilon}_{k'}$. Then,
%\begin{align*}
%C_k^{-\epsilon} &= \big(S_k^{+\epsilon} \cap M^c \big)^{-\epsilon} \cap \cX^{-\epsilon} = S_k \cap \big(M^c \big)^{-\epsilon} \cap \cX^{-\epsilon}.
%\end{align*}
%Now, $M^c = \cap_{k' \neq k} (S_k^{+\epsilon})^c$
Recall that $q_k(\cX^{-\epsilon}) = q(\cX) = 1$ by the support assumption. Hence, we have
\begin{gather*}
q_k(C_k^{-\epsilon}) \geq q_k\left( S_k \setminus \cup_{k' \neq k} S^{+2\epsilon}_{k'} \right) = q_k(S_k) - q_k \left(\cup_{k' \neq k} S^{+2\epsilon}_{k'} \right) \geq (1 - \delta) - \gamma \\
\implies 1 - q_k(C_k^{-\epsilon}) \leq \delta + \gamma.
\end{gather*}
Finally, as $\sum_k p_Y(y = k) = 1$, we have $R(f, \epsilon) \leq \delta + \gamma$ by convexity.
\end{proof}
\section{Proofs for \cref{escape}} \label{app:escape}
Let $\theta_0 \leq \pi/2$. We will show that the following classifier $f$ is robust in the setting of \cref{escape}:
\begin{gather*}
f(x) = \begin{cases}
1, &\text{ if } x^\top P \geq \theta_0 \\
2, &\text{otherwise}
\end{cases}.
\end{gather*}
Let $C_1 = \{x \in \bbS^{n-1} \colon x^\top P \geq \theta_0\}$ be the set of points classified into class $1$ by $f$. Similarly, let $C_2 = \bbS^{n-1} \setminus C_1$ be the set of points classified into class $2$. The robust risk of $f$ (measured w.r.t. perturbations in the geodesic distance $d$) can be expanded as
\begin{gather}
R_d(f, \epsilon) = 0.5 q_1(d^{+\epsilon}(C_2)) + 0.5 q_2(d^{+\epsilon}(C_1)), \label{example1risk}
\end{gather}
where $d^{+\epsilon}(S)$ denotes the $\epsilon$-expansion of the set $S$ under the distance $d$. Let $\epsilon \leq \theta_0$ (otherwise, the first term is $1/2$). Recall that the class conditional $q_1$ is defined as
\begin{gather*}
q_1(x) = \begin{cases}
\frac{1}{c_\psi} \frac{\psi(d(x, P))}{\sin^{n-2} d(x, P)}, &\text{ if } d(x, P) \leq \theta_0 \\
0, &\text{ otherwise }
\end{cases},
\end{gather*}
where $c_\psi$ is a normalizing constant which ensures that $q_1$ integrates to $1$, \ie, $c_\psi = \int_{\bbS^{n-1}} q_1$. Then, $q_2$ was defined to be uniform over the complement of the support of $q_1$, i.e. $q_2 = {\rm Unif}(\{x \in \bbS^{n-1} \colon d(x, P) > \theta_0\})$.
We can expand the first term of \cref{example1risk} as
\begin{align*}
q_1(d^{+\epsilon}(C_2))
&= q_1(d^{+\epsilon}(C_2) \setminus C_2) + q_1(C_2) \\
&= q_1(\{x \colon 0 < d(x, C_2) \leq \epsilon\}) + 0.
\end{align*}
Now for any $x \not \in C_2$, we have $d(x, C_2) = \theta_0 - d(x, P)$. Hence, $\{x \colon 0 < d(x, C_2) \leq \epsilon\} = \{x \colon 0.1 - \epsilon \leq d(x, P) < \theta_0\}$. Let $\theta(x)$ denote the angle that $x$ makes with $P$. Then, the geodesic distance is $d(x, P) = \theta(x)$. The earlier set is the same as the set of all points satisfying $\theta_0 - \epsilon \leq \theta(x) < 1$. This is nothing but the $\epsilon$-base of the hyper-spherical cap having angle $\theta_0$.
We continue,
\begin{align*}
q_1(\{x \colon \theta_0 - \epsilon < d(x, P) \leq \theta_0) &= q_1(\{x \colon \theta_0 - \epsilon \leq \theta(x) < \theta_0\})
\end{align*}
With a change of variables, the above can be evaluated as
\begin{align*}
q_1(\{\theta_0 - \epsilon \leq \theta(x) < \theta_0\})
&= \frac{1}{c_\psi} \int_{\theta_0 - \epsilon \leq \theta(x) < \theta_0} \frac{\psi(d(x, P))}{\sin(d(x, P))^{n-2}} dx\\
&= \frac{1}{c_\psi} \int^{\theta_0}_{\theta_0 - \epsilon} \frac{\psi(\theta)}{(\sin \theta)^{n-2}} \mu_{n-2}(\{ x \in \bbS^{n-1} \colon \theta(x) = \theta \}) d \theta,
\end{align*}
where $\mu_{n-2}$ is the $n-2$-dimensional volume ($n-1$-dimensional surface area). Now, $\mu_{n-2}(\{ x \in \bbS^{n-1} \colon \theta(x) = \theta \})$ is nothing but the volume of an slice of the hypersphere $\bbS^{n-1}$. This slice is in itself a hypersphere in $\R^{n-1}$, having radius $\sin \theta$. Thus, its volume is the same as the volume of the $\bbS^{n-2}$, scaled by $(\sin \theta)^{n-2}$. We continue,
\begin{align*}
\int^{\theta_0}_{\theta_0 - \epsilon} \frac{\psi(\theta)}{(\sin \theta)^{n-2}} \mu_{n-2}(\{ x \in \bbS^{n-1} \colon \theta(x) = \theta \}) d \theta
&= \int^{\theta_0}_{\theta_0 - \epsilon} \frac{\psi(\theta)}{(\sin \theta)^{n-2}} (\sin \theta)^{n-2} d \theta \\
&= \int^{\theta_0}_{\theta_0 - \epsilon} \psi(\theta) d\theta
\end{align*}
For illustration, we can take $\psi(\theta) = 1$, giving $q_1(d^{+\epsilon}(C_1)) = \epsilon / C$ from the above integral, with $c_\psi = \int_{0}^{\theta_0} \psi(\theta) d\theta = \theta_0 = \theta_0$.
We can now follow the same process as above, and expand the \cref{example1risk} as
\begin{gather*}
q_2(d^{+\epsilon}(C_1)) = \mu_{n-1}(\{x \colon \theta_0 \leq \theta(x) \leq \theta_0 + \epsilon),
\end{gather*}
where $\mu_{n-1}$ is again the $n-1$-dimensional volume. We use the following formula \cite{li2010concise} for the surface area of a hyperspherical cap (valid when $\alpha \
\leq \pi/2$):
\begin{gather*}
\mu_{n-1}(\{x \colon 0 \leq \theta(x) \leq \alpha\}) = m(\alpha) \overset{\rm def}{=} 0.5 \mu_{n-1}(\bbS^{n-1}) I_{\sin^2 \alpha}\left(\frac{n-1}{2}, \frac{1}{2}\right),
\end{gather*}
where $I(\cdot, \cdot)$ is the incomplete regularized beta function. Letting $d_{\theta_0} = m(\pi/2) + m(\pi/2) - m(\theta_0)$, we continue,
\begin{gather*}
q_2(d^{+\epsilon}(C_1)) = \begin{cases}
\frac{1}{d_{\theta_0}} \big(m(\theta_0 + \epsilon) - m(\theta_0)\big), &\text{ if } \epsilon + \theta_0 \leq \pi/2 \\
\frac{1}{d_{\theta_0}} \big(m(\pi/2) + m(\pi/2) - m(\theta_0 + \epsilon - \pi/2)\big), &\text{ if } \pi/2 \leq \epsilon + \theta_0 \leq \pi \\
\frac{1}{d_{\theta_0}}, &\text{ else }
\end{cases}.
% q_2(d^{+\epsilon}(C_1)) = 0.5 \left(I_{\sin^2 (0.1 + \epsilon)}\left(\frac{n-1}{2}, \frac{1}{2}\right) - I_{\sin^2 0.1}\left(\frac{n-1}{2}, \frac{1}{2}\right)\right)
\end{gather*}
%Hence, we have the robust risk of $f$ as
%\begin{gather}
% R_d(f, \epsilon) = 0.5 \min \left(\frac{\epsilon}{\theta_0}, 1\right) + 0.5 \left(I_{\sin^2 (0.1 + \epsilon)}\left(\frac{n-1}{2}, \frac{1}{2}\right) - I_{\sin^2 0.1}\left(\frac{n-1}{2}, \frac{1}{2}\right)\right). \label{sphereconcrisk}
%\end{gather}
%the second term in \eqref{sphereconcrisk} is much smaller than the first, and the above expression effectively behaves as $0.5 \epsilon / C$.
%For $\theta_0$ smaller than values of $\epsilon < \pi/2$, the risk from class $1$
The risk from class $1$ dominates till $\epsilon$ reaches around $\pi/2$, then the risk from class $2$ shoots up. We plot the resultant risk in \cref{fig:concplot1} (left).
\begin{figure}[h!]
\centering
\includegraphics[width=0.325\textwidth]{fig/sphere_conc_1}
\includegraphics[width=0.325\textwidth]{fig/sphere_conc_2}
\includegraphics[width=0.325\textwidth]{fig/sphere_conc_3}
\caption{Robust risk for classifiers for concentrated distributions on the sphere $\bbS^{n-1}$, for dimension $n = 100$. (Left): A plot of $R_d(f, \epsilon)$ w.r.t $p_1$. As $\epsilon$ grows from $0$ to $0.1$, class $1$ is the major contributor to the risk. At $\epsilon = 0.1$, all points in class $1$ admit an adversarial perturbation, taking them to class $2$. As $\epsilon$ grows further, the risk for class $2$ grows slowly, till $\epsilon$ reaches close to $\pi/2$, after which it blows up abruptly due to the high-dimensional fact that most of the mass of the uniform distribution lies near the equator of the sphere. (Middle): A plot of $R_d(g, \epsilon)$ w.r.t. $p_2$. As we make the support of the concentrated class larger, the distribution admits a classifier $g$ with slightly improved robustness for smaller $\epsilon$ values ($\leq 0.1$). (Right): A plot of $R_d(g, \epsilon)$ w.r.t. $p_3$: Finally, as both class conditionals are made concentrated, the classifier $g$ becomes quite robust.}
\label{fig:concplot1}
\end{figure}
Having presented the basic construction, we now comment on how \cref{escape} can be generalized. Firstly, we can vary the support of $q_1$, to have it cover a different fraction of the sphere. For instance, we can set $\theta_0 = \pi/2$ to get a modified $q_1$ as
\begin{gather*}
q'_1(x) = \begin{cases}
\frac{1}{C} \frac{1}{\sin^{n-2} d(x, P)}, &\text{ if } d(x, P) \leq \pi/2 \\
0, &\text{ otherwise }
\end{cases},
\end{gather*}
and let $q'_2$ be uniform over the complement of the support of $q'_1$. Along with balanced classes (\ie, $P(Y=1) = P(Y=2) = 1/2$), this gives the data distribution $p_2$. The robust classifier would now be given by the half-space $\{x \colon d(x, P) \leq \pi/2\}$. The risk over $p_2$ can be computed as earlier, and is plotted in \cref{fig:concplot1} (middle).
We can take this example further, and make both $q_1$ and $q_2$ concentrated. This can be done, for instance, by setting $q''_1 = q_1$, and setting $q''_2$ as follows:
\begin{gather*}
q''_2(x) = \begin{cases}
\frac{1}{C} \frac{1}{\sin^{n-2} d(x, -P)}, &\text{ if } d(x, -P) \leq \pi/2 \\
0, &\text{ otherwise }
\end{cases},
\end{gather*}
where $-P$ is the antipodal point of $P$. %The resultant data distribution $p_3$ is illustrated in \cref{fig:p3plot}.
As both class conditionals are now concentrated, the halfspace separating their supports becomes quite robust. This can be seen in the \cref{fig:concplot1} (right).
Lastly, $\psi$ can be taken to be a rapidly decaying function for even greater concentration and lesser robust risk, e.g., $\psi(\theta) = \exp (-\theta)$.
\section{Proofs for \cref{ex:subspaceconc}} \label{app:subspaceconc}
\subspaceconc*
It would be helpful to have \cref{fig:escape2_copy} in mind for what follows.
\begin{figure}[h!]
\centering
\includegraphics[width=0.6\textwidth]{fig/subspace_conc}
\caption{A plot of $q_1$ (orange), $q_2$ (violet) and the decision boundaries of $f$ (dashed).}
\label{fig:escape2_copy}
\end{figure}
We will demonstrate that the classifier $f$ illustrated in \cref{fig:escape2_copy} has low robust risk. We define $f$ for any $x \in B_\infty(0, 1)$, as follows (let $\gamma = e^{-\alpha}$):
\begin{gather*}
f(x) = \begin{cases}
1, &\text{ if } |x_1| \leq \gamma/2 + \epsilon, |x_2| \geq \gamma/2 + \epsilon \\
2, &\text{ if } |x_2| \leq \gamma/2 + \epsilon, |x_1| \geq \gamma/2 + \epsilon \\
1, &\text{otherwise}
\end{cases}.
\end{gather*}
The reason for splitting the cases for predicting $1$ into two subcases, is that we will only need to analyse the first subcase and the second will not contribute to the robust risk.
Defining $U_1$ to be the set of all points $x$ which admit an adversarial example for the class $1$, i.e. $U_1 = \{x \colon \exists \bar x \text{ such that } \|\bar x - x\|_2 \leq \epsilon, f(\bar x) = 2\}$. It is clear that
\begin{gather*}
U_1 \subseteq \{x \colon |x_1| \geq \gamma/2 \text{ or } |x_2| \leq \gamma/2 + 2\epsilon \}.
\end{gather*}
$U_2$ is defined analogously, and
\begin{gather*}
U_2 \subseteq \{x \colon |x_2| \geq \gamma/2 \text{ or } |x_1| \leq \gamma/2 + 2\epsilon \}.
\end{gather*}
Now, $R(f, \epsilon) = 0.5 q_1(U_1) + 0.5 q_2 (U_2)$. We look at the first term, and simplify an upper bound:
\begin{align}
q_1(U_1)
&\leq q_1(\{x \colon |x_1| \geq \gamma/2 \text{ or } |x_2| \leq \gamma/2 + 2\epsilon) \nonumber \\
&\leq q_1(\{x \colon |x_1| \geq \gamma/2\}) + q_1(\{x \colon |x_2| \leq \gamma/2+ 2\epsilon) \nonumber \\
&= 0 + q_1(\{ x \colon |x_2| \leq \gamma/2+ 2\epsilon \}). \label{q1upperbd}
\end{align}
Now, recall that $q_1 = {\rm Unif}(\{x \colon |x_1| \leq \gamma/2\})$, hence the expression in \eqref{q1upperbd} evaluates to
\begin{align*}
q_1(\{ x \colon |x_2| \leq \gamma/2+ 2\epsilon \})
&= \frac{1}{{\rm Vol}(\{x \colon |x_1| \leq \gamma/2\})} {\rm Vol}(\{ x \colon |x_2| \leq \gamma/2+ 2\epsilon, |x_1| \leq \gamma/2\}) \\
&= \frac{1}{2^{n-1} \gamma} 2^{n-2} \cdot (\gamma + 4\epsilon) \cdot \gamma \\
&= 0.5 \gamma + 2\epsilon
\end{align*}
The situation for $q_2$ is symmetric, and hence $q_2(U_2) = 0.5 (\gamma + 4\epsilon)$. Adding them together, we see that the robust risk is upper bounded by
\begin{gather*}
R(f, \epsilon) \leq 0.5 (e^{-\alpha} + 4\epsilon)
\end{gather*}
For the concentration parameters, note that the support of $q_1$, \ie, $S_1 = \{ x \colon |x_1| \leq \gamma/2, \|x\|_\infty \leq 1\}$ has volume $\gamma 2^{n-1} = 2^{n-1} \exp (-\alpha) = 0.5\ 2^n \exp (-n (\alpha / n)) \leq 0.5 \exp (-n (\alpha / n - 1))$. Hence, $q_1$ is $(0.5, \alpha/n - 1, 0)$ concentrated over $S_1$. Similarly, $q_2$ is $(0.5, \alpha/n - 1, 0)$-concentrated over $S_2 = \{ x \colon |x_2| \leq \gamma/2, \|x\|_\infty \leq 1\}$. Finally,
\begin{align*}
q_1(S_2^{+2\epsilon})
&= q_1(\{\|x\|_\infty \leq 1, \|x_1\| \leq \gamma/2, \|x_2\| \leq \gamma/2 + 2 \epsilon\}) \\
&= 2^{n-2} \gamma (\gamma + 4 \epsilon) \gamma^{-1} 2^{1-n} \\
&= 0.5 \gamma + 2\epsilon
\end{align*}
\section{Proof of \cref{lem:noisyl2}} \label{app:noisyl2}
\noisyl*
\begin{proof}
Let $\lambda \x' \in C(\x)$ as defined in \eqref{eq:conenoisyl2}. There exist $\f \in F(\x), \v \in V(\x)$ such that $\lambda \x' = \f + \v$. We will show that the projection of $\lambda \x'$ onto $F$ is given by $\f$. Recall that
\begin{equation}
A_\lambda(\x') = \{\bt_i \colon \langle \bt_i, \d^*_{\lambda}(\x') \rangle = 1\}, \text{ where } \d^*_\lambda(\x') = {\rm Proj}_{K^\circ}(\lambda \x').
\end{equation}
Choose any $\z \in K^\circ$. Recall that ${\rm Proj}_{K^\circ}(\lambda \x') = \min_{\z \in K^\circ} \|\z - \lambda \x'\|_2$. Consider the objective,
\begin{equation}
\|\z - \lambda \x'\|_2^2 = \|(\z - \f) - \v\|_2^2 = \|\z - \f\|_2^2 + \|\v\|_2^2 - 2 \langle \z - \f, \v \rangle. \label{eq:distance}
\end{equation}
We will show that $\langle \z - \f, \v \rangle$ is negative. Consider any $\bt_i \in A_\lambda(\x)$, and observe that $\langle \z, \bt_i \rangle \leq 1$,
%\rene{Why do we need $s_i$. AP: Fixed}
as $\z \in K^\circ$. However, from the definition of $F(\x)$ recall that $\langle \f, \bt_i \rangle = 1$. This implies that $\langle \z - \f, \bt_i \rangle \leq 0$. Using the fact that $\v \in V(\x)$, we expand the inner product as
\begin{equation}
\langle \z - \f, \v \rangle = \sum_{\bt_i \in A_\lambda(\x)} \alpha_i \langle \z - \f, \bt_i \rangle \leq 0. \label{eq:obtuseangle}
\end{equation}
Using \eqref{eq:obtuseangle} in \eqref{eq:distance} to obtain
\begin{equation}
\|\z - \lambda \x'\|_2^2 \geq \|\z - \f\|_2^2 + \|\v\|_2^2 \geq 0.
\end{equation}
The minimizer above is obtained at $\z = \f \in C(\x)$, and hence we have shown that for all $\lambda \x' \in C(\x)$, ${\rm Proj}_{K^\circ}(\lambda \x') = \f$. Finally, $\d^*_\lambda(\x') = {\rm Proj}_{K^\circ}(\lambda \x') = \f \in F(\x)$. The theorem statement then follows from the definition of $F(\x)$.
\end{proof}
\section{Further Experimental Details and Qualitative Examples} \label{app:expt}
\subsection{Comparison along Projection on $C_\lambda$}
In this section, we will provide more details on attacking a classifier with perturbations that lie on our certified set $C_\lambda$. %To recall,
\paragraph{Details of the Projection Step \rom{2} in \eqref{pgdmod}}
Recall from \cref{lem:noisyl2} that $C_\lambda(x)$ is defined as the Minkowski sum of the face $F_\lambda(x)$ \eqref{face} and the cone $V_\lambda(x)$ \eqref{cone}. Given an iterate $x^t$, we can compute its projection onto $C_\lambda(x^0)$ by solving the following optimization problem
\begin{align}
{\rm Proj}_{C_\lambda(x^0)} (x^t)
&= \argmin_x \|x^t - x\|_2 \text{ s.t. } x \in C_\lambda(x^0) \nonumber
\end{align}
Now every $x \in C_\lambda(x^0)$ can be written as $x = d + v$ where $d \in F(x^0), v \in V(x^0)$. $d$ is such that $t^\top d = 1$ for all $t_i \in A_\lambda(x^0)$, and $t^\top d \leq 1$ otherwise. Then, $v$ is such that $v = \sum_{t_i \in A_\lambda(x^0)} \alpha_i t_i$ for some $\alpha_i \geq 0$. Recall that $T=[t_1, t_2, \ldots, t_{2M}]$ denotes the matrix containing the training data-points as well as their negations. Let the matrix $A_1 \in \R^{d \times |A_\lambda(x^t)|}$ contain all the columns of $T$ in $A_\lambda(x^t)$ and let $A_2 \in \R^{d \times (2M - |A_\lambda(x^t)|)}$ contain all the remaining columns. Then the objective of the optimization problem above can be written as
\begin{align}
\min_{\alpha, d} \left \|x^t - A_1 \alpha - d \right \|^2_2 \text{ s.t. } A_1^\top d = \1, A_2^\top d \leq \1, \alpha \geq \0 \label{qp}
\end{align}
\eqref{qp} is a linearly constrained quadratic program, and can be solved by standard optimization tools. In particular, we use the \texttt{qp} solver from the python \texttt{cvxopt} library.
\paragraph{Visualization of the Certified Set}
In \cref{fig:support}, we visualize the set $A_\lambda(x)$, \ie, the set of active constraints at the optimal solution of the dual problem, for several $x$ in the MNIST test set. It can be seen that $A_\lambda(x)$ contains images of the same digit as $x$ in most cases. This is expected, as taking the majority label among $A_\lambda(x)$ gives an accuracy of around $97\%$ (solid blue curve in \cref{fig:comparison1}). Note that all these images $t \in A_\lambda(x)$ are also contained in the certified set, $C_\lambda(x)$, which can be seen as $t = t + \0$, with $t \in F_\lambda(x)$, and $\0 \in V_\lambda(x)$.
\begin{figure}[h]
\centering
%\includegraphics[width=0.49\textwidth]{fig/support_vis_1.pdf}
%\includegraphics[width=0.49\textwidth]{fig/support_vis_2.pdf}
\includegraphics[width=\textwidth]{fig/support_vis.pdf}
\caption{A visualization of the set of active constraints $A_\lambda(x)$ for several $x$. Atmost $5$ members of $A_\lambda(x)$ are shown for every $x$.}
\label{fig:support}
\end{figure}
\paragraph{Visualization of Attacks restricted to the Certified Set}
In \cref{fig:projattackvis}, we visualize the attacks restricted to our certified set $C_\lambda$ computed by \eqref{pgdmod} for the classifier $g^{\rm RS}_{0.02}$. We observe that we can identify the correct digit from most of the attacked images, but the randomized smoothing classifier is incorrect at higher $\epsilon$. A small number of these attacked images are close to the actual class decision boundary, and the class is ambigous. This is expected, both due to the inherent class ambiguity present in some MNIST images, as well as the large $\epsilon$ we are attacking with. For all these images, the prediction of our classifier $g_\lambda$ is certified to be accurate. For comparison, the RS certificate is unable to certify anything beyond $\epsilon \geq 0.06$.
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{fig/proj_attack_vis.pdf}
\caption{A visualization of attacks on the RS classifier $g^{\rm RS}_{0.02}$ restricted to our certified set $C_\lambda$ obtained by \eqref{pgdmod}. Different rows plot different attack strengths $\epsilon$. Whenever an image is misclassified, the red annotation on the top left shows the class predicted by $g^{\rm RS}_{0.02}$.}
\label{fig:projattackvis}
\end{figure}
%
\subsection{Comparison along $\ell_2$ balls}
\paragraph{Exact Certification for $g_\lambda$}
In this section, we first state and prove \cref{lem:noisyl2} that allows us to exactly compute the $\ell_2$ certified radius for our classifier $g_\lambda$ at any point $x$.
\begin{lemma}
For all $\x' \in C(\x)$ as defined in \cref{lem:noisyl2}, we have $g_\lambda(\x') = g_\lambda(\x)$. Additionally, for all $\| \v \|_2 \leq r_0(\x)$ we have $g_\lambda(\x + \v) = g_\lambda(\x)$, where
\begin{align}
\begin{split}
r_0(\x) =
&\quad\;\; \min_{\u} \quad -\langle \x, \u \rangle \\
&\quad \text{sub. to} \quad
%\begin{aligned}&
\|\u\|_2 = 1, \quad
%\\
%&
\langle \u, \bt_i \rangle \leq 0 \ \forall \bt_i \in A_\lambda(\x), \quad
%\\
%&
\langle \u, \v \rangle \leq 1 \ \forall \v \in {\rm ext}(F(\x)),
%\end{aligned}
\end{split} \label{eq:optexact}
\end{align}
where ${\rm ext}(F(\x))$ is the set of extreme points of the polyhedron $F(\x)$.
\label{lem:gcertnoisy}
\end{lemma}
\paragraph{Discussion} Before presenting the proof, let us ponder over the result. We see that \eqref{eq:optexact} involves solving an optimization problem having as many constraints as the number of extreme points of the polyhedron $F(x)$. In general, this can be very large in high dimensions, and hence computationally inefficient to compute. As a result, while \cref{lem:gcertnoisy} provides an exact certificate in theory, it is hard to use in practice without further approximations.
\begin{proof}
Recall that given a set $A_\lambda(\x)$, the polyhedron $C(\x)$ is defined in \eqref{eq:conenoisyl2} as
\begin{equation}
C(\x) = F(\x) + V(\x),
\end{equation}
where $F(\x)$ is a polyhedron, and $V(\x)$ is a cone given as the conic hull of $A_\lambda(\x)$, \ie, $V(\x) = \text{cone}(A_\lambda(\x))$. We want to find the size $r(\x)$ of the largest $\ell_2$ ball centered at $\x$, \ie, $B(\x, r(\x))$, that can be inscribed within $C(\x)$.
From convex geometry \cite{basu2019introduction}, we know that any polyhedron can be described as an intersection over halfspaces obtained from its polar, $C^\circ$ as
\begin{equation}
C(\x) = \cap_{\u \in C^\circ(\x)} H^-(\u, 0), \label{eq:halfspace}
\end{equation}
where $H^-(\u, 0) = \{\x \colon \langle \u, \x \rangle \leq 0\}$. The distance of $\x$ from the boundary of $C(\x)$ is the same as the smallest distance of $\x$ from any halfspace in \eqref{eq:halfspace}. However, this distance is simply
\begin{equation}
\text{dist}(\x, H^-(\u, 0)) = -\left \langle \x, \frac{\u}{\|\u\|_2} \right \rangle. \label{eq:dist}
\end{equation}
In other words, we can expressing $C(\x)$ as the intersection over halfspaces whose normals lie in $C^\circ(\x)$, \ie,
\begin{equation}
r(\x) = \left( \min_{\u} -\langle \x, \u \rangle, \text{ sub. to } \|\u\|_2 = 1, \u \in C^\circ(\x) \right).\label{eq:link1}%
\end{equation}
All that is left is to obtain a description of $C^\circ(\x)$. This can be done by first expressing $C(\x)$ in a standard form, as
\begin{equation}
C(\x) = \text{conv}(\text{ext}(F(\x))) + \text{cone}(A_\lambda(\x)),
\end{equation}
where $\text{ext}(F(\x))$ denotes the extreme points of the polyhedron $F(\x)$, $\text{conv}(\cdot)$ denotes the convex hull and $\text{cone}(\cdot)$ denotes the conic hull. Now we can apply a theorem in convex geometry to obtain the polar (\cite[Th. 2.79]{basu2019introduction}) as
\begin{equation}
C^\circ(\x) = \left \{\u \colon \begin{array}{l} \langle \u, \bt_i \rangle \leq 0 \ \forall \bt_i \in A_\lambda(\x)\\ \langle \u, \v \rangle \leq 1 \ \forall \v \in \text{ext}(F(\x)) \end{array} \right \} \label{eq:link2}
\end{equation}
Combining \eqref{eq:link1} and \eqref{eq:link2}, we obtain the lemma statement. Finally, we note that as \cref{lem:noisyl2} shows that $A_\lambda(\x') = A_\lambda(\x)$ for all $\lambda \x' \in C(\x)$, and the classifier $g_\lambda(\x)$ is purely a function of $A_\lambda(\x)$, we have that $g_\lambda(\x') = g_\lambda(\x)$ for all $\lambda \x' \in C(\x)$.
\end{proof}
\paragraph{Black Box Attacks on $g_\lambda$} Since our classifier $g_\lambda$ is not explicitly differentiable with respect to its input, we use the HopSkipJump \cite{chen2020hopskipjumpattack} black-box attack to obtain adversarial perturbations that cause a misclassification. The obtained attacked images, and the pertubation magnitude are shown in \cref{fig:blackbox}.
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{fig/blackbox_vis.pdf}
\caption{A visualization of attacks on our classifier $g_\lambda$ obtained by the HopSkipJump black box attack \cite{chen2020hopskipjumpattack}. The label predicted by $g_\lambda$ is shown as a red annotation on the top left. The $\ell_2$ perturbation magnitude $\epsilon$ is shown at the bottom. Note that the $\epsilon$ values are not nice fractions because they are set by the HopSkipJump attack using a binary search to find the minimum attack strength that causes a misclassification.}
\label{fig:blackbox}
\end{figure}
%
\subsection{Details of Randomized Smoothing Certificates}
For obtaining the certified accuracy curves using Randomized Smoothing \cite{cohen2019certified} (solid lines in \cref{fig:comparison1,fig:comparison2}),
we follow the certification and prediction algorithms for RS in \cite[Sec 3.2.2]{cohen2019certified}: given a base classifier $f$ and a test image $x$, we use $n_0 = 100$ samples from an isotropic gaussian distribution $\mathcal{N}(0,\sigma)$ to prediction the majority class $g_\sigma^{\rm RS}(x)$. We then use $n = 100$ samples to estimate the prediction probability $\underline{p_A}$ under $\mathcal{N}(0, \sigma)$ with confidence atleast $0.999$. If $\underline{p_A}$ is at least $0.5$, we report the certified radius $\sigma \Phi^{-1}(\underline{p_A})$. Otherwise, if $\underline{p_A} < 0.5$, we abstain and return a certified radius of $0$.
%\subsection{Miscellaneous Experimental Details}
\paragraph{Evaluating the Dual Classifier $g_\lambda$} We normalize each image in the MNIST dataset to have unit $\ell_2$ norm, and resize to $32 \times 32$. We pick $10000$ images $\s_i$ at random from the training set to obtain our data matrix $\S \in \R^{1024 \times 10000}$. We now pick a test point $\x$, and then solve the dual problem \eqref{eq:dualnoise} with $\lambda = 2$, to obtain the dual solution $\d^*_\lambda(\x)$. This allows us to then obtain the set of active constraints $A_\lambda(\x)$ in \eqref{eq:activelambda}. Then, we use the majority rule as the aggregate function in \eqref{eq:g} to obtain the classfication output from the dual classifier $g_\lambda(\x)$. We sample $100$ and $500$ images $x$ uniformly at random from the MNIST test set for generating the curves in \cref{fig:comparison1,fig:comparison2}, respectively. All experiments are performed on a NVIDIA GeForce RTX 2080 Ti GPU with 12 GB memory.
\subsection{Discussion of Computational Cost}
The computational complexity of obtaining the certificate $C(x)$ in \cref{lem:noisyl2} is dominated by solving the linear optimization problem \eqref{eq:optisproj} to obtain the set $A_\lambda$, which has $n + M$ variables. It is known that in practice, the cost of solving LPs is much lower than the worst case \cite{spielman2004smoothed}, and it takes us $11.7$ seconds on average on a single CPU for each image $x$ without parallel processing.
For comparison to other certified defenses (like Randomized Smoothing), we perform $T = 20$ steps of \eqref{pgdmod}, and each Step II of \eqref{pgdmod} requires solving a quadratic optimization program, given in \eqref{qp}. This is a linearly constrained quadratic program. In practice it takes us $18.1$ seconds on average on a single CPU for each $x$ without parallel processing.
\subsection{Experiments on CIFAR-10} \label{app:cifar10}
We provide experiments for our method applied to CIFAR-10. To do this, we first embed each CIFAR-10 image into a feature space, designed such that $\ell_2$-bounded perturbations in the feature space, i.e., $\phi(x + v), \|v\|_2 \leq \epsilon$, correspond to semantic perturbations in the input space, e.g., distorting the image. Such a feature space is obtained following recent popular work in learning perceptual metrics in vision \cite{zhang2018unreasonable}, where the task is precisely to learn a feature representation where $\ell_2$ distances align with human perception. We now use our subspace model on $\phi(X)$, and perform exactly the experiments in \cref{sec:expt}, and make similar observations: we obtain reasonable robust accuracy using our method (\cref{fig:reb1} blue lines), and this accuracy is maintained within our certified polyhedra at high $\epsilon$, where existing defenses are not robust.
Though such experiments showing feature space certificates have appeared in the literature \cite{voravcek2022provably}, the question of whether the perceptual representation is itself susceptible to small adversarial perturbations still remains for future exploration.
\begin{figure}[h!]
\centering
\includegraphics[width=0.4\textwidth]{fig/rebuttal_projected_attacks.pdf} \hspace{4em}
\includegraphics[width=0.4\textwidth]{fig/rebuttal_projected_attacks_SENet.pdf}
\caption{Comparing Randomized Smoothing (RS) with Our Method from \cref{sec:subspace}, for adversarial perturbations computed by repeating Steps I, II from \cref{pgdmod}. This figure is the counterpart of \cref{fig:comparison1}, where the MNIST dataset has been changed to $\phi($CIFAR-10$)$, and the transformation $\phi$ is given by (left) $384$-dimensional slice of LPIPS \cite{zhang2018unreasonable} features (i.e., output of the third Conv2D layer) and (right) $128$-dimensional input features used by SENet \cite{zhang2021learning}. Observation 1: Our method retains robustness even when the dashed empirical upper bound for robust accuracy for the RS classifier drops to random chance. Observation 2: Our robust accuracy (solid blue line) gets higher from left to right, as the feature space of SENet is closer to a union of subspaces than LPIPS. On the other hand, the left feature space aligns better with human perception than the right feature space.}
\label{fig:reb1}
\end{figure}
\subsection{Evaluating \cref{th:strongconc} on Empirical Distributions} \label{app:cifar10lbcomparison}
The empirical results in \cite{pydi2020adversarial,bhagoji2019lower} create an empirical distribution $\hat p$ by selecting two classes from CIFAR-10, which simply places probability mass $1 / N$ on each of the $N$ samples in the dataset. \cite{pydi2020adversarial,bhagoji2019lower} evaluate their lower bounds on $\hat p$, and not on the actual real world distribution $p$, which might be arbitrarily complex. Hence, it is unclear whether the trends observed hold for $p$. Nevertheless, the same experiment can be translated to our setting for any real world dataset, our \cref{th:strongconc} would then show the existence of a robust classifier for $\hat p$. We would need to solve a discrete optimization problem for finding the concentrated sets in \cref{th:strongconc}. Even though valid only for $\hat p$, this analysis is interesting, and is reported in \cref{fig:reb2}.
\begin{figure}[h]
\centering
\includegraphics[width=0.4\textwidth]{fig/conc_eval_cifar10.pdf}
\caption{Instantiating \cref{th:strongconc} for the CIFAR-10 empirical distribution. We construct the empirical distribution $\hat p$ which assigns equal probability to each $(x, y)$ pair in the CIFAR-10 test set. This leads to empirical class conditionals $\hat q_1, \ldots, \hat q_{10}$, corresponding to the $10$ classes. We set $\gamma = 0$ in \cref{th:strongconc}, and greedily construct the sets $S_1, \ldots, S_{10}$: sort all images in decreasing order of their distance to their nearest neighbor in the test set, pick the first $m$ images, and then compute the concentration parameters (minimum separation $\epsilon_m$ and mass $\delta_m = 1 - \min_i \hat q_i(S_i)$). We then plot $(\epsilon_m, 1 - \delta_m)$ for several choices of $m$. Applying \cref{th:strongconc}, each $(\epsilon_m, \delta_m)$ guarantees the existence of a $(\epsilon_m, \delta_m)$-robust classifier for $\hat p$.}
\label{fig:reb2}
\end{figure}
\section{Discussion on Strong Concentration}
\label{app:strongconcdisc}
\paragraph{Are Natural Image Distributions Strongly Concentrated?} Consider the task of classifying images of dogs vs cats. Any small $\ell_2$ perturbation (say of size $\epsilon = 0.1$) to an image of a dog is not likely to change it to an image of a cat -- the true label (or a human decision) does not change with small perturbations. Let $S_{\rm cat}$ be the set of images that are cats with a high confidence, and similarly for
$S_{\rm dog}$. What we just said was that any image in the
$\epsilon$-expansion of $S_{\rm cat}$ is not likely to be a dog, and vice versa. This is precisely the condition of separation in the definition of strong concentration: $q_{\rm dog}(S_{\rm cat}^{+\epsilon}) \leq \gamma$, and $q_{\rm cat}(S_{dog}^{+\epsilon}) \leq \gamma$, for a small $\gamma$.
Now, despite the above, the task of practically classifying an
image into a dog or a cat is not trivial at all, as the sets $S_{\rm dog}$ and $S_{\rm cat}$ might be very complex, and hence it might be computationally hard to find a predictor (neural network, or any other classifier) for distinguishing $S_{\rm dog}$ and $S_{\rm cat}$. Using modern deep learning, one is able to learn an approximation to these sets that leads to small standard error, but this approximation is still bad enough that the learned predictor is very susceptible to adversarial examples. This is to say that task of robustly classifying an image into dog or cat is even harder.
\paragraph{Tightness of Strong Concentration}
In \cref{def:strongconc}, we are trying to obtain a sufficient condition for the existence of a robust classifier. There are two parts to \cref{def:strongconc}: the concentration, and the separation conditions. The first is essential, as \cref{th:concentration} asserts that whenever a robust classifier exists for a data distribution, the class conditionals are concentrated (i.e., it is necessary).
As we argued above, the separation condition is not very strong, and should be satisfied by image distributions like MNIST, CIFAR-10 and ImageNet. Let us walk through this condition to see if it can be further relaxed. As we note at the start of \cref{sec:sufficientcond}, if all the class conditionals were to be concentrated on subsets $S_k$ that have high intersection among each other, then even benign classification would be hard (i.e., benign risk will be high), let alone robust classification. So, even for the existence of a good benign classifier, it is essential for $S_k \cap S_{k'}$ to be close to empty, for all $k \neq {k'}$.
Now, even if $S_k \cap S_{k'}$ were almost empty, for a robust classifier, we care about the $\epsilon$-expansions of these sets to not intersect. In other words, we do not want an $\epsilon$-perturbed cat to look like a dog. Hence, for the existence of a good robust classifier, $S_k \cap S_{k'}^{+\epsilon}$ should be close to empty, for all $k \neq k'$. This can be generalized in measure terms, to require $q_k(S_{k'}^{+\epsilon}) \leq \gamma$ , for a small $\gamma$. Upto this, all these conditions are essential for obtaining a robust classifier.
Now, note that in \cref{def:strongconc}, the expansion is taken for $2\epsilon$, instead of $\epsilon$. This is where our proof technique for \cref{th:strongconc} incurs a slack of $\epsilon$, and we believe a different approach for constructing the robust classifier might be able to reduce the expansion requirement to $\epsilon$. This could be an interesting future avenue for relaxing \cref{def:strongconc} and thereby strengthening \cref{th:strongconc}.
%{\color{red} mention 100 test images}
%\emph{Evaluating the Certificate $C(\x)$}. Once we have obtained $A_\lambda(\x)$ as above, we can obtain $C(\x) = F(\x) + V(\x)$ as defined in \eqref{eq:conenoisyl2}. Note that $\lambda \x \in C(\x)$, and by the construction of $C$, we have $\lambda \x + V(\x) \subseteq C(\x)$, and hence \cref{lem:noisyl2} guarantees robust classification for any perturbation $\x' \in \lambda \x + V(\x)$. In \cref{fig:detres}, we pick a particular $\v \in V(\x)$, for each $\x$, and demonstrate that $g_\lambda(\x) = g_\lambda(\x + \alpha \v)$ for arbitrary $\alpha > 0$. Specifically, we pick $\v = \bt_i$ for some $\bt_i \in A_\lambda (\x)$. \cref{fig:detres} extends \cref{fig:my_label} in the main text by providing more examples.
%
%\emph{Comparison to Existing Certificates}. Observe that in \cref{fig:detres}, we certify perturbations that are \emph{not} bounded by $\ell_2$ norm, in contrast to much of the existing literature. Instead, we are able to certify perturbations where the true class remains unchanged, but the actual image changes significantly in appearance.
%\section{Ambiguities in Primal Solutions}
%Recall the primal problem in \eqref{eq:l1}, \ie, $ \min_\c \| \c \|_1 \text{ sub. to } \x = \S \c$. It is well known that there might be multiple solutions to \eqref{eq:l1}, all with different supports. We showed an example of such ambiguities in \cref{sec:noiseless}. We demonstrate another pathological training dataset here, \ie, $\S = [\s_1, \s_2, -\s_1, -\s_2]$, with $\s_1 \neq \s_2$. Consider a test-point $\x = \s_1 + \s_2$, and notice that there are several (in fact, infinitely many) solutions to \eqref{eq:l1}, all having the same optimal value, e.g., $\c = [1, 1, 0, 0], [0, 0, -1, -1], [0.3, 1, -0.7, 0]$, etc. These solutions have different supports, and it is unclear how to obtain a robust classifier by looking at the support. To get around this ambiguity, we move to the dual problem in \cref{sec:noiseless}, which ultimately allows us to obtain a robust dual classifier.
% \include{deprecated}
% \include{deprecated2}
%\section{Proof of \cref{lem:gcertefficient}} \label{sec:proof1}
%\begin{proof}
%Recall that given a set $A(\x)$, the cone $C(\x)$ is defined in \eqref{eq:cone} as
%\begin{equation}
%C(\x) = \left\{ \sum_{\bt_i \in A(\x)} \alpha_i \bt_i \colon \alpha_i > 0 \right\}.
%\end{equation}
%We want to find the size $r(\x)$ of the largest $\ell_2$ ball centered at $\x$, \ie, $B(\x, r(\x))$, that can be inscribed within $C(\x)$. From convex geometry \cite{basu2019introduction}, we know that any cone can be described as an intersection over halfspaces obtained from its dual cone, $C^\circ$ as
%\begin{equation}
%C(\x) = \cap_{\u \in C^\circ(\x)} H^-(\u, 0), \label{eq:halfspace}
%\end{equation}
%where $H^-(\u, 0) = \{\x \colon \langle \u, \x \rangle \leq 0\}$. Hence, in the distance of $\x$ from the boundary of $C(\x)$ is the same as the smallest distance of $\x$ from any cone in \eqref{eq:halfspace}. However, this distance is simply
%\begin{equation}
%\text{dist}(\x, H^-(\u, 0)) = -\left \langle \x, \frac{\u}{\|\u\|_2} \right \rangle. \label{eq:dist}
%\end{equation}
%Additionally, it is known that given the generators $A(\x)$ of the cone $C(\x)$, the polar set can be equivalently written as $C^\circ(\x) = \{ \u \colon \langle \u, \bt_i \rangle \leq 0 \ \forall \bt_i \in A(\x) \}$ \cite{basu2019introduction}. Combining with \eqref{eq:dist}, this shows that
%\begin{equation}
%r(\x) = \left( \min_{\u} -\langle \x, \u \rangle, \text{ sub. to } \|\u\|_2 = 1, \langle \u, \bt_i \rangle \leq 0 \ \forall \bt_i \in A(\x) \right).%
%\end{equation}%
%For completeness, we note that as \cref{lem:noiseless} shows that $A(\x') = A(\x)$ for all $\x' \in C(\x)$, and the classifier $g(\x)$ is purely a function of $A(\x)$, we have that $g(\x') = g(\x)$ for all $\x' \in C(\x)$.
%\end{proof}
%\section{Rebuttal: Visualization of the Certified Region $C(\x)$}
%In \cref{fig:reb1,fig:reb2,fig:reb3}, we demonstrate that the directions in the certified cone $C(\x)$ can be visually dissimilar.
%\begin{figure}[h]
% \centering
% \includegraphics[width=\textwidth]{fig/rebuttal/supp_example_2}
% \includegraphics[width=\textwidth]{fig/rebuttal/unnorm_example_2}
% \caption{Top Two Rows: Obtained Directions in the certificate $C(\x)$, Bottom Row: $\x + \alpha \v$ for some $\v \in C(\x)$, and $\alpha > 0$.}
% \label{fig:reb1}
%\end{figure}
%
%\begin{figure}[h]
% \centering
% \includegraphics[width=\textwidth]{fig/rebuttal/supp_example_0}
% \includegraphics[width=\textwidth]{fig/rebuttal/unnorm_example_0}
% \caption{Top Two Rows: Obtained Directions in the certificate $C(\x)$, Bottom Row: $\x + \alpha \v$ for some $\v \in C(\x)$, and $\alpha > 0$.}
% \label{fig:reb2}
%\end{figure}
%
%\begin{figure}[h]
% \centering
% \includegraphics[width=\textwidth]{fig/rebuttal/supp_example_8}
% \includegraphics[width=\textwidth]{fig/rebuttal/unnorm_example_8}
% \caption{Top Two Rows: Obtained Directions in the certificate $C(\x)$, Bottom Row: $\x + \alpha \v$ for some $\v \in C(\x)$, and $\alpha > 0$.}
% \label{fig:reb3}
%\end{figure}
%\begin{figure}[h]
% \centering
% \includegraphics[width=\textwidth]{fig/rebuttal/supp_example_3}
% \includegraphics[width=\textwidth]{fig/rebuttal/unnorm_example_3}
% \caption{Top Row: Obtained Directions in the certificate $C(\x)$, Bottom Row: $\x + \alpha \v$ for some $\v \in C(\x)$, and $\alpha > 0$.}
% \label{fig:reb4}
%\end{figure}
%
%$\qquad$
%\vspace{15cm}
\end{document}
\begin{thebibliography}{10}
\bibitem{carliniA}
Anish Athalye, Nicholas Carlini, and David Wagner.
\newblock Obfuscated gradients give a false sense of security: Circumventing
defenses to adversarial examples.
\newblock In {\em International conference on machine learning}, pages
274--283. PMLR, 2018.
\bibitem{awasthi2021adversarially}
Pranjal Awasthi, Vaggos Chatziafratis, Xue Chen, and Aravindan Vijayaraghavan.
\newblock Adversarially robust low dimensional representations.
\newblock In {\em Conference on Learning Theory}, pages 237--325. PMLR, 2021.
\bibitem{awasthi2020adversarial}
Pranjal Awasthi, Himanshu Jain, Ankit~Singh Rawat, and Aravindan
Vijayaraghavan.
\newblock Adversarial robustness via robust low rank representations.
\newblock {\em Advances in Neural Information Processing Systems},
33:11391--11403, 2020.
\bibitem{basu2019introduction}
Amitabh Basu.
\newblock Introduction to convexity.
\newblock {\em Optimization}, 52:37, 2019.
\bibitem{bhagoji2019lower}
Arjun~Nitin Bhagoji, Daniel Cullina, and Prateek Mittal.
\newblock Lower bounds on adversarial robustness from optimal transport.
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{bubeck}
S{\'e}bastien Bubeck, Yin~Tat Lee, Eric Price, and Ilya Razenshteyn.
\newblock Adversarial examples from computational constraints.
\newblock In {\em International Conference on Machine Learning}, pages
831--840. PMLR, 2019.
\bibitem{cappelli2002subspace}
Raffaele Cappelli, Dario Maio, and Davide Maltoni.
\newblock Subspace classification for face recognition.
\newblock In {\em International Workshop on Biometric Authentication}, pages
133--141. Springer, 2002.
\bibitem{carliniB}
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas
Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey
Kurakin.
\newblock On evaluating adversarial robustness.
\newblock {\em arXiv preprint arXiv:1902.06705}, 2019.
\bibitem{chen2020hopskipjumpattack}
Jianbo Chen, Michael~I Jordan, and Martin~J Wainwright.
\newblock Hopskipjumpattack: A query-efficient decision-based attack.
\newblock In {\em 2020 ieee symposium on security and privacy (sp)}, pages
1277--1294. IEEE, 2020.
\bibitem{chiang2020certified}
Ping-yeh Chiang, Renkun Ni, Ahmed Abdelkader, Chen Zhu, Christoph Studer, and
Tom Goldstein.
\newblock Certified defenses for adversarial patches.
\newblock In {\em 8th International Conference on Learning Representations
(ICLR 2020)(virtual)}. International Conference on Learning Representations,
2020.
\bibitem{cohen2019certified}
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter.
\newblock Certified adversarial robustness via randomized smoothing.
\newblock In {\em International Conference on Machine Learning}, pages
1310--1320. PMLR, 2019.
\bibitem{dohmatob}
Elvis Dohmatob.
\newblock Generalized no free lunch theorem for adversarial robustness.
\newblock In {\em International Conference on Machine Learning}, pages
1646--1654. PMLR, 2019.
\bibitem{donoho2003optimally}
David~L Donoho and Michael Elad.
\newblock Optimally sparse representation in general (nonorthogonal)
dictionaries via l1 minimization.
\newblock {\em Proceedings of the National Academy of Sciences},
100(5):2197--2202, 2003.
\bibitem{eirasancer}
Francisco Eiras, Motasem Alfarra, Philip Torr, M~Pawan Kumar, Puneet~K Dokania,
Bernard Ghanem, and Adel Bibi.
\newblock Ancer: Anisotropic certification via sample-wise volume maximization.
\newblock {\em Transactions of Machine Learning Research}.
\bibitem{elhamifar2011robust}
Ehsan Elhamifar and Ren{\'e} Vidal.
\newblock Robust classification using structured sparse representation.
\newblock In {\em CVPR 2011}, pages 1873--1879. IEEE, 2011.
\bibitem{elhamifar2012block}
Ehsan Elhamifar and Ren{\'e} Vidal.
\newblock Block-sparse recovery via convex optimization.
\newblock {\em IEEE Transactions on Signal Processing}, 60(8):4094--4107, 2012.
\bibitem{humanadversarials}
Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey
Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein.
\newblock Adversarial examples that fool both computer vision and time-limited
humans.
\newblock {\em Advances in neural information processing systems}, 31, 2018.
\bibitem{fawzi}
Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi.
\newblock Adversarial vulnerability for any classifier.
\newblock {\em Advances in neural information processing systems}, 31, 2018.
\bibitem{fischer2020certified}
Marc Fischer, Maximilian Baader, and Martin Vechev.
\newblock Certified defense to image transformations via randomized smoothing.
\newblock {\em Advances in Neural Information Processing Systems},
33:8404--8417, 2020.
\bibitem{foucart2013invitation}
Simon Foucart and Holger Rauhut.
\newblock An invitation to compressive sensing.
\newblock In {\em A mathematical introduction to compressive sensing}, pages
1--39. Springer, 2013.
\bibitem{gilmer2018relationship}
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel~S Schoenholz, Maithra Raghu,
Martin Wattenberg, Ian Goodfellow, and G~Brain.
\newblock The relationship between high-dimensional geometry and adversarial
examples.
\newblock {\em arXiv preprint arXiv:1801.02774}, 2018.
\bibitem{guo2018countering}
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van~der Maaten.
\newblock Countering adversarial images using input transformations.
\newblock In {\em International Conference on Learning Representations}, 2018.
\bibitem{hui2012sparse}
Kang-hua Hui, Chun-li Li, and Lei Zhang.
\newblock Sparse neighbor representation for classification.
\newblock {\em Pattern Recognition Letters}, 33(5):661--669, 2012.
\bibitem{jin2020manifold}
Charles Jin and Martin Rinard.
\newblock Manifold regularization for locally stable deep neural networks.
\newblock {\em arXiv preprint arXiv:2003.04286}, 2020.
\bibitem{Kaba:ICML21}
Mustafa~D Kaba, Chong You, Daniel~P Robinson, Enrique Mallada, and Rene Vidal.
\newblock A nullspace property for subspace-preserving recovery.
\newblock In Marina Meila and Tong Zhang, editors, {\em Proceedings of the 38th
International Conference on Machine Learning}, volume 139 of {\em Proceedings
of Machine Learning Research}, pages 5180--5188. PMLR, 18--24 Jul 2021.
\bibitem{khoury2018geometry}
Marc Khoury and Dylan Hadfield-Menell.
\newblock On the geometry of adversarial examples.
\newblock {\em arXiv preprint arXiv:1811.00525}, 2018.
\bibitem{laaksonen1997subspace}
Jorma Laaksonen et~al.
\newblock {\em Subspace classifiers in recognition of handwritten digits}.
\newblock Helsinki University of Technology, 1997.
\bibitem{lecun}
Yann LeCun.
\newblock The mnist database of handwritten digits.
\newblock {\em http://yann.lecun.com/exdb/mnist/}, 1998.
\bibitem{levine2020randomized}
Alexander Levine and Soheil Feizi.
\newblock (de) randomized smoothing for certifiable defense against patch
attacks.
\newblock {\em Advances in Neural Information Processing Systems},
33:6465--6475, 2020.
\bibitem{li2018geometric}
Chun-Guang Li, Chong You, and Ren{\'e} Vidal.
\newblock On geometric analysis of affine sparse subspace clustering.
\newblock {\em IEEE Journal of Selected Topics in Signal Processing},
12(6):1520--1533, 2018.
\bibitem{li2010concise}
Shengqiao Li.
\newblock Concise formulas for the area and volume of a hyperspherical cap.
\newblock {\em Asian Journal of Mathematics \& Statistics}, 4(1):66--70, 2010.
\bibitem{li2013nearest}
Wei Li, Eric~W Tramel, Saurabh Prasad, and James~E Fowler.
\newblock Nearest regularized subspace for hyperspectral classification.
\newblock {\em IEEE Transactions on Geoscience and Remote Sensing},
52(1):477--489, 2013.
\bibitem{lin2020dual}
Wei-An Lin, Chun~Pong Lau, Alexander Levine, Rama Chellappa, and Soheil Feizi.
\newblock Dual manifold adversarial robustness: Defense against lp and non-lp
adversarial attacks.
\newblock {\em Advances in Neural Information Processing Systems},
33:3487--3498, 2020.
\bibitem{ma2018characterizing}
Xingjun Ma, Bo~Li, Yisen Wang, Sarah~M Erfani, Sudanthi Wijewickrema, Grant
Schoenebeck, Dawn Song, Michael~E Houle, and James Bailey.
\newblock Characterizing adversarial subspaces using local intrinsic
dimensionality.
\newblock In {\em International Conference on Learning Representations}, 2018.
\bibitem{mahloujifar2019curse}
Saeed Mahloujifar, Dimitrios~I Diochnos, and Mohammad Mahmoody.
\newblock The curse of concentration in robust learning: Evasion and poisoning
attacks from concentration of measure.
\newblock In {\em Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~33, pages 4536--4543, 2019.
\bibitem{mahloujifar2019empirically}
Saeed Mahloujifar, Xiao Zhang, Mohammad Mahmoody, and David Evans.
\newblock Empirically measuring concentration: Fundamental limits on intrinsic
robustness.
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{moosavi2019robustness}
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal
Frossard.
\newblock Robustness via curvature regularization, and vice versa.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 9078--9086, 2019.
\bibitem{moosavi2018divide}
Seyed-Mohsen Moosavi-Dezfooli, Ashish Shrivastava, and Oncel Tuzel.
\newblock Divide, denoise, and defend against adversarial attacks.
\newblock {\em arXiv preprint arXiv:1802.06806}, 2018.
\bibitem{naseem2010linear}
Imran Naseem, Roberto Togneri, and Mohammed Bennamoun.
\newblock Linear regression for face recognition.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
32(11):2106--2112, 2010.
\bibitem{niu2020limitations}
Zhonghan Niu, Zhaoxi Chen, Linyi Li, Yubin Yang, Bo~Li, and Jinfeng Yi.
\newblock On the limitations of denoising strategies as adversarial defenses.
\newblock {\em arXiv preprint arXiv:2012.09384}, 2020.
\bibitem{pal2023understanding}
Ambar Pal and Jeremias Sulam.
\newblock Understanding noise-augmented training for randomized smoothing.
\newblock {\em Transactions on Machine Learning Research}, 2023.
\bibitem{papernot2016distillation}
Nicolas Papernot, Patrick McDaniel, Xi~Wu, Somesh Jha, and Ananthram Swami.
\newblock Distillation as a defense to adversarial perturbations against deep
neural networks.
\newblock In {\em 2016 IEEE symposium on security and privacy (SP)}, pages
582--597. IEEE, 2016.
\bibitem{somayeh}
Samuel Pfrommer, Brendon~G Anderson, and Somayeh Sojoudi.
\newblock Projected randomized smoothing for certified adversarial robustness.
\newblock 2022.
\bibitem{pydi2020adversarial}
Muni~Sreenivas Pydi and Varun Jog.
\newblock Adversarial risk via optimal transport and optimal couplings.
\newblock In {\em International Conference on Machine Learning}, pages
7814--7823. PMLR, 2020.
\bibitem{smoothadv}
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien
Bubeck, and Greg Yang.
\newblock Provably robust deep learning via adversarially trained smoothed
classifiers.
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{samangouei2018defense}
Pouya Samangouei, Maya Kabkab, and Rama Chellappa.
\newblock Defense-gan: Protecting classifiers against adversarial attacks using
generative models.
\newblock In {\em International Conference on Learning Representations}, 2018.
\bibitem{madryB}
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and
Aleksander Madry.
\newblock Adversarially robust generalization requires more data.
\newblock {\em Advances in neural information processing systems}, 31, 2018.
\bibitem{goldstein}
Ali Shafahi, W~Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein.
\newblock Are adversarial examples inevitable?
\newblock {\em arXiv preprint arXiv:1809.02104}, 2018.
\bibitem{shafahi2019adversarial}
Ali Shafahi, Mahyar Najibi, Mohammad~Amin Ghiasi, Zheng Xu, John Dickerson,
Christoph Studer, Larry~S Davis, Gavin Taylor, and Tom Goldstein.
\newblock Adversarial training for free!
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{soltanolkotabi2012geometric}
Mahdi Soltanolkotabi and Emmanuel~J Candes.
\newblock A geometric analysis of subspace clustering with outliers.
\newblock {\em The Annals of Statistics}, 40(4):2195--2238, 2012.
\bibitem{soltanolkotabi2014robust}
Mahdi Soltanolkotabi, Ehsan Elhamifar, and Emmanuel~J Candes.
\newblock Robust subspace clustering.
\newblock {\em The annals of Statistics}, 42(2):669--699, 2014.
\bibitem{spielman2004smoothed}
Daniel~A Spielman and Shang-Hua Teng.
\newblock Smoothed analysis of algorithms: Why the simplex algorithm usually
takes polynomial time.
\newblock {\em Journal of the ACM (JACM)}, 51(3):385--463, 2004.
\bibitem{stutz2019disentangling}
David Stutz, Matthias Hein, and Bernt Schiele.
\newblock Disentangling adversarial robustness and generalization.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 6976--6987, 2019.
\bibitem{sulam2020adversarial}
Jeremias Sulam, Ramchandran Muthukumar, and Raman Arora.
\newblock Adversarial robustness of supervised sparse coding.
\newblock {\em Advances in neural information processing systems},
33:2110--2121, 2020.
\bibitem{szegedy2013intriguing}
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
Ian Goodfellow, and Rob Fergus.
\newblock Intriguing properties of neural networks.
\newblock {\em arXiv preprint arXiv:1312.6199}, 2013.
\bibitem{madryA}
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and
Aleksander Madry.
\newblock Robustness may be at odds with accuracy.
\newblock {\em arXiv preprint arXiv:1805.12152}, 2018.
\bibitem{renebook}
Ren{\'e} Vidal, Yi~Ma, S~Shankar Sastry, Ren{\'e} Vidal, Yi~Ma, and S~Shankar
Sastry.
\newblock {\em Generalized Principal Component Analysis}.
\newblock Springer, 2016.
\bibitem{voravcek2022provably}
V{\'a}clav Vor{\'a}cek and Matthias Hein.
\newblock Provably adversarially robust nearest prototype classifiers.
\newblock In {\em International Conference on Machine Learning}, pages
22361--22383. PMLR, 2022.
\bibitem{wang2013noisy}
Yu-Xiang Wang and Huan Xu.
\newblock Noisy sparse subspace clustering.
\newblock In {\em International Conference on Machine Learning}, pages 89--97.
PMLR, 2013.
\bibitem{wong2019fast}
Eric Wong, Leslie Rice, and J~Zico Kolter.
\newblock Fast is better than free: Revisiting adversarial training.
\newblock In {\em International Conference on Learning Representations}, 2019.
\bibitem{wright2010sparse}
John Wright, Yi~Ma, Julien Mairal, Guillermo Sapiro, Thomas~S Huang, and
Shuicheng Yan.
\newblock Sparse representation for computer vision and pattern recognition.
\newblock {\em Proceedings of the IEEE}, 98(6):1031--1044, 2010.
\bibitem{wright2008robust}
John Wright, Allen~Y Yang, Arvind Ganesh, S~Shankar Sastry, and Yi~Ma.
\newblock Robust face recognition via sparse representation.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
31(2):210--227, 2008.
\bibitem{yang2020randomized}
Greg Yang, Tony Duan, J~Edward Hu, Hadi Salman, Ilya Razenshteyn, and Jerry Li.
\newblock Randomized smoothing of all shapes and sizes.
\newblock In {\em International Conference on Machine Learning}, pages
10693--10705. PMLR, 2020.
\bibitem{you2015geometric}
Chong You and Ren{\'e} Vidal.
\newblock Geometric conditions for subspace-sparse recovery.
\newblock In {\em International Conference on Machine Learning}, pages
1585--1593. PMLR, 2015.
\bibitem{yu2015semi}
Guoxian Yu, Guoji Zhang, Zili Zhang, Zhiwen Yu, and Lin Deng.
\newblock Semi-supervised classification based on subspace sparse
representation.
\newblock {\em Knowledge and Information Systems}, 43(1):81--101, 2015.
\bibitem{zhang2022rethinking}
Bohang Zhang, Du~Jiang, Di~He, and Liwei Wang.
\newblock Rethinking lipschitz neural networks for certified l-infinity
robustness.
\newblock {\em arXiv preprint arXiv:2210.01787}, 2022.
\bibitem{zhang2018unreasonable}
Richard Zhang, Phillip Isola, Alexei~A Efros, Eli Shechtman, and Oliver Wang.
\newblock The unreasonable effectiveness of deep features as a perceptual
metric.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 586--595, 2018.
\bibitem{zhang2021learning}
Shangzhi Zhang, Chong You, Ren{\'e} Vidal, and Chun-Guang Li.
\newblock Learning a self-expressive network for subspace clustering.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 12393--12403, 2021.
\bibitem{zhang2021manifold}
Shufei Zhang, Kaizhu Huang, Jianke Zhu, and Yang Liu.
\newblock Manifold adversarial training for supervised and semi-supervised
learning.
\newblock {\em Neural Networks}, 140:282--293, 2021.
\bibitem{zhang2021understanding}
Xiao Zhang and David Evans.
\newblock Understanding intrinsic robustness using label uncertainty.
\newblock {\em arXiv preprint arXiv:2107.03250}, 2021.
\end{thebibliography}