Detection of Adversarial Physical Attacks in Time-Series Image Data
Extracted source text · arXiv 2304.13919
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\newtheorem{problem}{Problem}
% Recommended, but optional, packages for figures and better typesetting:https://www.overleaf.com/project/5e37990b63ee6f0001cca9a5
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{hyperref}
\usepackage{url} % simple URL typesetting
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
%\usepackage{microtype} % microtypography
\usepackage{graphicx} % for pdf, bitmapped graphics files
\graphicspath{{figures/}}
%\usepackage{hyperref}
\usepackage{epsfig} % for postscript graphics files
\usepackage{ragged2e}
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{epstopdf}
%\usepackage{cite}
\usepackage{algorithm}
\usepackage{algorithmic}
%\usepackage[noend,ruled,linesnumbered]{algorithm2e}
%\SetKwComment{Comment}{$\triangleright$\ } {}
\usepackage{multirow}
\usepackage{rotating}
\usepackage{subfigure}
\usepackage{color}
\usepackage{mysymbol}
\usepackage{url}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{cor}[theorem]{Corollary}
\newtheorem{lem}[theorem]{Lemma}
\newtheorem{prop}[theorem]{Proposition}
\newtheorem{assumption}[theorem]{Assumption}
\newtheorem{pr}{Problem}
%\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
%\theoremstyle{remark}
\newtheorem{rem}[theorem]{Remark}
\newtheorem{ex}[theorem]{Example}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}
\title{Detection of Adversarial Physical Attacks in Time-Series Image Data\\
%{\footnotesize \textsuperscript{*}Note: Sub-titles are not captured in Xplore and should not be used}
\thanks{$^*$Authors with equal contribution.}
}
\makeatletter
\newcommand{\linebreakand}{%
\end{@IEEEauthorhalign}
\hfill\mbox{}\par
\mbox{}\hfill\begin{@IEEEauthorhalign}
}
\makeatother
\author{
\IEEEauthorblockN{Ramneet Kaur$^*$}
\IEEEauthorblockA{\textit{Computer and Information Science} \\
\textit{University of Pennsylvania}\\
Philadelphia, PA, USA \\
ramneetk@seas.upenn.edu}
\and
\IEEEauthorblockN{Yiannis Kantaros$^*$}
\IEEEauthorblockA{\textit{Electrical and Systems Engineering} \\
\textit{Washington University in St. Louis}\\
St. Louis, MO, USA \\
ioannisk@wustl.edu}
\and
\IEEEauthorblockN{Wenwen Si$^*$}
\IEEEauthorblockA{\textit{Computer and Information Science} \\
\textit{University of Pennsylvania}\\
Philadelphia, PA, USA \\
wenwens@seas.upenn.edu}
\linebreakand
\IEEEauthorblockN{James Weimer}
\IEEEauthorblockA{\textit{Computer Science} \\
\textit{Vanderbilt University}\\
Nashvilee, TN, USA \\
james.weimer@vanderbilt.edu}
\and
\IEEEauthorblockN{Insup Lee}
\IEEEauthorblockA{\textit{Computer and Information Science} \\
\textit{University of Pennsylvania}\\
Philadelphia, PA, USA \\
lee@cis.upenn.edu}
}
\maketitle
\begin{abstract}
Deep neural networks (DNN) have become a common sensing modality in autonomous systems as they allow for semantically perceiving the ambient environment given input images. Nevertheless, DNN models have proven to be vulnerable to adversarial digital and physical attacks. To mitigate this issue, several detection frameworks have been proposed to detect whether a \textit{single} input image has been manipulated by adversarial \textit{digital} noise or not. In our prior work, we proposed a real-time detector, called VisionGuard (VG), for adversarial physical attacks against single input images to DNN models. Building upon that work, we propose $\text{VisionGuard}^*$ ($\text{VG}^*$), that couples VG with majority-vote methods, to detect adversarial physical attacks in \textit{time-series image data}, e.g., videos. This is motivated by autonomous systems applications where images are collected over time using onboard sensors for decision making purposes. We emphasize that majority-vote mechanisms are quite common in autonomous system applications (among many other applications), as e.g., in autonomous driving stacks for object detection. In this paper, we investigate, both theoretically and experimentally, how this widely used mechanism can be leveraged to enhance performance of adversarial detectors. We have evaluated $\text{VG}^*$ on videos of both clean and physically attacked traffic signs generated by a state-of-the-art robust physical attack. We provide extensive comparative experiments against detectors that have been designed originally for out-of-distribution data and digitally attacked images.
\end{abstract}
\begin{IEEEkeywords}
adversarial detectors, adversarial examples, neural networks, image classification
\end{IEEEkeywords}
\section{Introduction}\label{sec:introduction}
\input{Intro}
\section{Problem Formulation}\label{sec:attacks}
\input{ProblemFormulation}
\section{$\text{VisionGuard}^*$: A New Adversarial Time-Series Detector }\label{sec:ImageDetector}
\input{ProposedDetector.tex}
\section{Experiments}\label{sec:sim1}
\input{Experiments}
\vspace{-0.2cm}
\section{Conclusion}%\textcolor{red}{[to be updated]}
\vspace{-0.1cm}
This paper proposed an effective and modular defense against physical adversarial attacks in time series image data. %The proposed defense mechanism builds upon exist single-image detectors and majority vote frameworks.
We provided conditions under which by using historical data, the time series detector performs better than the corresponding single image detector. Finally, we provided extensive experiments to validate the proposed time-series detector.
\input{Appendix}
\bibliography{YK_bib}
\bibliographystyle{IEEEtran}
\end{document}
%!TEX root = ../main.tex
%\section{Introduction}\label{sec:introduction}
%Deep learning has demonstrated impressive performance across a wide range of applications such as object recognition \cite{he2016deep}, image classification \cite{krizhevsky2012imagenet}, language processing \cite{kumar2016ask}, and others.
%\textcolor{red}{[OLD]}
Deep neural networks (DNN) have seen renewed interest in the last decade due to the vast amount of available data and recent advances in computing. In autonomous systems, DNN are typically used as feedback controllers, motion planners, or perception modules such as object detection and classification.
Nevertheless, brittleness of DNN has resulted in unreliable behavior and public failures, e.g., Tesla cars' crashes and Uber running a red light \cite{uber_crash}. %\textcolor{red}{RK: I have added~\citep{uber_crash} as a citation to the Uber crashing into pedestrian case..we can add that or any other citation here}.
In fact, several adversarial attack methods have been proposed recently that aim to minimally manipulate inputs to DNN models to cause desired incorrect outputs that would benefit an attacker. In image classification, that is also considered in this paper, adversarial attacks are typically categorized as digital and physical ones. The latter augment the physical world by placing adversarial stickers on the surface of objects of interest \cite{karmon2018lavan,li2019adversarial,eykholt2018robust,brown2017adversarial,boloor2020attacking} (see e.g., Fig. \ref{fig:X}), while the former embeds an imperceptible amount of digital noise to the classifier input data \cite{carlini2017adversarial,papernot2016limitations,choi2022argan}.
Digital attacks typically assume a threat model in which the adversary can feed data directly into the machine learning component. This is not necessarily the case for systems operating in the physical world, such as mobile robots or driverless cars that are using signals from cameras and other sensors as input. Motivated by these applications, this paper focuses exclusively on adversarial physical attacks.
%Although adversarial digital noise is imperceptible...
%Deep neural networks (DNN) have been deployed in multiple safety-critical systems, such as medical imaging, autonomous
%cars, and surveillance systems. At the
%same time, DNN have been shown to be vulnerable to adversarial examples~\cite{szegedy2013intriguing}, i.e., inputs which have deliberately been modified to cause either misclassification or desired incorrect prediction that would benefit an attacker. Adversarial examples in the literature can be divided into two sub-classes depending on how the attack is executed. One augments the physical environment to induce misclassification (e.g., adding a sticker to a stop sign) \cite{karmon2018lavan,eykholt2018robust}, while the other adds a small perturbation to the classifier input data; see Figures \ref{fig:attack_example_p}-\ref{fig:attack_example_d}. %In this work, when we refer to adversarial examples, we only refer to the latter subclass (i.e., small perturbation attacks).
%https://www.overleaf.com/project/5f775c9ac6077c00013f2187
%Adversarial examples, especially in the case of image classification that is also considered in this paper, have received increased research attention due to the following properties. First, the difference between legitimate and adversarial digital inputs can be imperceptible, making adversarial detection a very challenging task~\cite{szegedy2013intriguing}. Second, the transferability of adversarial digital samples between different models allows for black-box attacks~\cite{goodfellow2014explaining,zantedeschi2017efficient}. Third, the robustness of physical attacks against various environmental conditions and backgrounds \cite{eykholt2018robust}. Fourth, both digital and physical adversarial samples are often misclassified with high confidence, implying that DNN fail to discriminate between adversarial and legitimate inputs~\cite{guo2017calibration,eykholt2018robust}.
%However, these defenses typically rely on training new classifiers and, therefore, are also vulnerable to adversarial inputs and can be easily bypassed \cite{papernot2016transferability,gilmer2018adversarial,carlini2017adversarial,carlini2017magnet}.
%
%\TC{all of the above works rely on building separate classifiers?}
%To protect DNN-based image classifiers from adversarial attacks, several detection and monitoring methods have recently been proposed. These methods identify whether a given input image has been manipulated or not. As it will be discussed in Section \ref{sec:related}, the majority of existing detectors focuses on \textit{single} images that may have been under \textit{digital} attacks.
Several \textit{detectors} have been proposed recently to detect adversarially attacked images; see Section \ref{sec:related}. Nevertheless, the large majority of these works primarily focus on determining whether a single image has been manipulated adversarially by a digital attack or not. To the contrary, in this paper, we focus on \textit{detecting} adversarial \textit{physical} attacks against DNN for image classification tasks in \textit{time-series image data} (e.g., videos).
%
%
Our work is motivated by autonomous driving applications, where a time-ordered sequence of images, collected by onboard cameras, containing objects of interest is available that is used for control purposes. For instance, consider an autonomous car collecting images of a stop sign as it approaches an intersection. Our goal is to determine if the objects (e.g., traffic signs) in the collected images have been manipulated by adversarial targeted physical attacks proposed in \cite{eykholt2018robust}; examples of attacked traffic signs using \cite{eykholt2018robust} are shown in Fig. \ref{fig:X}.
%
%
%Particularly, we consider a state-of-art robust physical attack designing targeted adversarial stickers that can yield desired misclassifications once they are placed on the surface of an object of interest \cite{eykholt2018robust}. Examples of attacked traffic signs are shown in Fig. \ref{fig:X}.
%VG is a real-time and attack-agnostic detector against both digital and physical attacks. %VG does not modify the specific classifier and, does not rely on building separate classifiers. %; as such, the proposed approach complements existing defenses against adversarial examples such as adversarial/robust training and image purification.
%Instead,
%The key idea in VG is that adversaries may be successful at fooling DNN due to the large feature space over which they can look for adversarial inputs. Motivated by this, VG aims to shrink the feature space available to adversaries. In particular, to determine if an image is adversarial, VG checks if the softmax output of the target classifier on a given input image changes significantly after feeding it a refined version of that image. To refine images, i.e., to squeeze out possibly unnecessary features of the input image, label-invariant transformations are employed (e.g., brightness transformations). Then, similarity of the corresponding softmax outputs using the Kullback–Leibler (K-L) divergence metric is measured. If this metric is above a threshold, the image is classified as adversarial; otherwise, it is classified as clean.
%
%
%Second, motivated by autonomous driving and mobile robotics applications where a sequence of measurements/images are collected over time for decision making purposes \cite{kantaros2020perception,bansal2020combining}, we propose a new detector, called $\text{VisionGuard}^*$ ($\text{VG}^*$), to detect \textit{physical} attacks in \textit{time-series image data}. For instance, consider a car approaching a stop sign or a robot encountering a person. Typically, in these applications, a sequence of images is collected, which are classified by on-board perception systems for control and decision-making purposes. %In particular, this paper considers robust physical attacks that generate adversarial stickers being placed on the surface of objects of interest (e.g., traffic signs) \cite{eykholt2018robust}.
%To design a time-series detector, we build upon existing adversarial (single-image) detectors and majority vote mechanisms.
%
%$\text{VG}^*$ relies on coupling VG with existing majority-vote mechanisms.
In particular, we consider the following problem. We assume that a time-ordered sequence of images containing a single object of interest is given. We also assume the existence of a perfect object tracking algorithm that is capable of creating an ordered sequence containing the same object. This assumption can be relaxed by using e.g., recent probabilistic data association algorithms that can associate semantic sensor measurements to objects; see e.g., \cite{bowman2017probabilistic}. %\IL{May need to elaborate how, instead of just stating out of the scope.} %Then,
Our goal is to determine whether the object captured in the time-series data has been adversarially manipulated \cite{eykholt2018robust}. To achieve this, we design a new, simple yet effective, time-series detector, called $\text{VisionGuard}^*$ ($\text{VG}^*$). The proposed time-series detector builds upon (i) VisionGuard (VG), a recently proposed single-image detector against various digital attacks and the physical attacks \cite{kantaros2021real}, and (ii) majority vote mechanisms. Specifically, $\text{VG}^*$ consists of the following two steps: (1) VG is applied to all images that lie within a sliding detection window (capturing how many past images will be used) classifying each image as either adversarial or clean; (2) majority-vote is applied over the VG outputs. If the majority of the VG outputs is `adversarial', then $\text{VG}^*$ classifies the object under investigation as adversarial; otherwise, it is considered clean.
%
%We argue that exploiting past observations of an object interest, instead of relying only on the current input image, can enhance performance of adversarial detectors resulting in safer perception-based control loops.
%
Intuitively, the detection performance of $\text{VG}^*$ should increase as the size of available data increases or, equivalently, the length of the sliding window increases. To formalize this intuition, we provide conditions about the length of the sliding window and the detection accuracy of VG, under which the time-series detector achieves better detection performance than the single-image detector. In other words, this result shows that, as expected, exploiting past observations of an object interest - instead of relying only on the current input image - can enhance performance of adversarial detectors resulting in safer perception-based control loops.
%
We emphasize that majority-vote mechanisms are quite common in autonomous system applications (among several other applications), as e.g., in autonomous driving stacks \cite{Apollo} for object detection. Our goal is to investigate how this widely used mechanism can be leveraged to enhance performance of adversarial detectors. %In this paper, we argue that exploiting past observations of an object interest, instead of relying only on the current input image, can enhance performance of adversarial detectors resulting in safer perception-based control loops.
%
We demonstrate the efficiency of $\text{VG}^*$ on videos of clean and physically attacked traffic signs using stickers designed as in \cite{eykholt2018robust}.
Additionally, we highlight that VG can be replaced by any other single-image detector to construct a time-series detector. Particularly, in our experiments, we compare the performance of $\text{VG}^*$ against time-series detectors constructed by simply replacing VG in step (1) with the detectors developed in \cite{lee2018simple,liang2018detecting}.
%
We note that \cite{lee2018simple} proposes a detector for digitally attacked and out-of-distribution (OOD) data, while \cite{liang2018detecting} focuses exclusively on digital attacks.
%
Our experiments validate the previously theoretical connection between the length of detection window and the accuracy of the employed single-image detector, i.e., the more accurate the single-image detector, the better the performance of the corresponding time-series detector as the window length increases. %Finally, we note again that in our evaluation, for simplicity we assume that each video contains a single object; otherwise, object tracking and data association algorithms need to be employed to generate sequences of images for each detected object \cite{bowman2017probabilistic}.
Finally, in the appendix, we provide new comparative experiments of VG against the detectors in \cite{lee2018simple,liang2018detecting} on adversarial patches \cite{brown2017adversarial} that do not exist in the original paper that proposed VG \cite{kantaros2021real}. %\textcolor{red}{RK: We can replace the citation~\cite{kantaros2021real} by VG to be more clear}.
The adversarial attack in \cite{brown2017adversarial} generates patches that can be placed
in the vicinity of an object of interest to cause desired misclassifications. %This attack is also effective by digitally embedding the sticker in the background of given images.
%
Our comparative experiments further validate the claim made in \cite{kantaros2021real} that existing detectors that have been originally developed for digitally attacked or OOD data, as e.g., \cite{lee2018simple,liang2018detecting}, may perform poorly at detecting physical attacks.
%In this paper, we extend this work in the following two ways.
%
%First, we evaluate the detection performance of VG against both adversarial stickers generated by \cite{eykholt2018robust} on traffic signs as well as adversarial patches \cite{brown2017adversarial} placed in the background of ImageNet images \cite{deng2009imagenet}. Also, we provide additional comparative experiments against existing detectors \cite{lee2018simple,liang2018detecting} that do not exist in \cite{kantaros2021real}. Particularly, \cite{lee2018simple} designs a detector for digitally attacked and OOD data, while \cite{liang2018detecting} focuses exclusively on digital attacks. %Our comparative experiments suggest that \textcolor{red}{+++}
%Our comparative experiments demonstrate that existing detectors that have been originally developed for digitally attacked or out-of distribution data, as e.g., \cite{lee2018simple,liang2018detecting}, may perform poorly at detecting physical attacks.
%and we have employed three recently proposed single-image adversarial detectors \cite{kantaros2021real,lee2018simple,liang2018detecting}. We note that \cite{kantaros2021real} proposes a detector for both digital and physical attacks, \cite{lee2018simple} designs a detector for digitally attacked and OOD data, while \cite{liang2018detecting} focuses exclusively on digital attacks.
%We provide extensive comparisons of both VG and $\text{VG}^*$ and existing recently proposed detectors for adversarial attacks and OOD-data. Particularly, we compare the proposed detectors against \cite{lee2018simple,liang2018detecting}. designs a detector for digitally attacked and OOD data, while \cite{liang2018detecting} focuses exclusively on digital attacks.
%the single-image detectors and the resulting time-series detectors and we validate the previously theoretical connection between the length of detection window and the detection accuracy.
%Our experiments demonstrate the real-time requirements of the time-series detectors along with their high-detection performance when certain mild conditions are met.
%
%Our experiments also demonstrate that existing single-image detectors that have been originally developed for digitally attacked or out-of distribution data, as e.g., \cite{lee2018simple,liang2018detecting}, may perform poorly at detecting physical attacks on certain objects of interest. In fact, we show that the detector in \cite{kantaros2021real} that was originally developed for physical attacks outperforms \cite{lee2018simple,liang2018detecting} in both the single-image and the time-series context. Finally, we note that in our evaluation, for simplicity we assume that each video contains a single object; otherwise, object tracking and data association algorithms need to be employed to generate sequences of images for each detected object \cite{bowman2017probabilistic}.
%Each video contains a single traffic sign and every image frame of the video is sent to an (imperfect) object detector trained to detect traffic signs. Frames that include the traffic signs are by time-series detector. %Our experiments show that the larger the sliding detection window is, the better the detection performance.
%and we show, as expected, that their performance can be enhanced by leveraging historical data, i.e., past images containing the same object of interest.
%To establish reliability and security of DNN-based perception systems against adversarial input images, we propose, VisionGuard (VG), a novel attack- and dataset-agnostic detection framework. VG does not modify the specific classifier and, does not rely on building separate classifiers. %; as such, the proposed approach complements existing defenses against adversarial examples such as adversarial/robust training and image purification.
%Instead, VG relies on the observation that adversaries may be successful at fooling DNN due to the large feature space over which they can look for adversarial inputs. This is also validated in our experiments: the larger the input space (i.e., image dimensions), the easier to fool the target classifier. Motivated by this, the proposed defense aims to shrink the feature space available to adversaries. In particular, to determine if an image is adversarial, VG checks if the softmax output of the target classifier on a given input image changes significantly after feeding it a refined version of that image. To refine images, i.e., to squeeze out possibly unnecessary features of the input image, we employ label-invariant transformations. Specifically, our experiments suggest that lossy compression (e.g., JPEG) with high compression quality and brightness/rotation transformations can achieve high detection performance against digital and physical adversarial attacks, respectively; however, any other transformation can be integrated with the proposed detector. %\IL{Not clear what the following sentence means "however, ... seamlessly." Incorportated to where?}
%lossy compression algorithms (e.g., JPEG) with high compression quality.
%Then, we measure the similarity of the corresponding softmax outputs using the Kullback–Leibler (K-L) divergence metric. If this metric is above a threshold, the image is classified as adversarial; otherwise, it is classified as clean. %We would like to highlight that the proposed detector is general enough in the sense that alternative transformation can be incorporated seamlessly.
%Finally, we would like to emphasize that in this paper we are particularly interested in real-time applications (e.g., autonomous cars) imposing runtime constraints on both the attacker and defender that are typically ignored in the related literature. In particular, the attacker has to be capable of generating adversarial noise at a rate that is greater than the rate at which images are received and classified by the target perception system while the detector has to reason about the trustworthiness of input images at a rate that is smaller than the attack rate. As a result, we focus on attacks that are either fast enough to craft at runtime (e.g., defense-oblivious digital attacks such as FGSM and CW \cite{carlini2017towards} %\IL{Isn't CW white box? YK: CW is still a black box attack in the sense that it doesn't take into account the defense/detector.})
%or physical attacks that are generated offline but they can fool perception systems at runtime \cite{eykholt2018robust}. Observe that this precludes white-box attacks - that assume perfect knowledge of both the target DNN and the defense mechanism - due to their high computational cost. Nevertheless, such attacks do not concern current users, since it is almost impossible to get such complete information \cite{chen2019damagenet}.
%\begin{figure}[t]
% \centering
% %\includegraphics[width=0.40\linewidth]{accuracyNN_vs_attacks_MNIST_CIFAR10.pdf}}
% %\includegraphics[width=1\linewidth]{figures/CWatt2.png}
% \vspace{-0.2cm}
% \includegraphics[width=0.6\linewidth]{stickers.png}
% \caption{Examples of adversarial stickers that fool the LISA-CNN (left) \cite{eykholt2018robust} and GTSRB-CNN (right) \cite{stallkamp2012man} so that the stop sign is missclassified as speed limit 35 sign; picture borrowed from \cite{eykholt2018robust}.\vspace{-0.4cm}}
% \label{fig:attack_example_p}
%\end{figure}
\begin{figure}[t]
\centering
\subfigure{
\label{fig:stop2}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.30\linewidth]{stop2.jpg}}
\subfigure{
\label{fig:yield1}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.30\linewidth]{yield2.jpg}}
\subfigure{
\label{fig:speed2}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.23\linewidth]{speed35.jpg}}
\vspace{-0.35cm}
%\subfigure[Transformed Clean Images]{
% \label{fig:accTransformed}
% \includegraphics[width=0.32\linewidth]{accuracyNN_transformedTestSet.png}}
\caption{%Figures \ref{fig:acc1MNIST} and \ref{fig:acc2MNIST} show the
Traffic signs manipulated adversarially by the $\text{RP}_2$ attack \cite{eykholt2018robust} to fool LISA-CNN. The stickers have been placed so that the stop, yield and speed limit signs are classified as
speed limit 35, speed limit 35, and turn right signs, respectively \cite{kantaros2021real}.\vspace{-0.6cm}
%as per the perturbation shown in Figure \ref{fig:attackedl1}.
}
\label{fig:X}
\end{figure}
%\begin{figure}[t]
% \centering
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_attacks_MNIST_CIFAR10.pdf}}
%\includegraphics[width=1\linewidth]{figures/CWatt2.png}
% \includegraphics[width=1\linewidth]{CWatt.png}
% \caption{Examples of almost imperceptible adversarial images from ImageNet after CW attack.}
% \label{fig:attack_example_d}
%\end{figure}
%\paragraph{Related Works:}
% \vspace{-0.3cm}
\subsection{Related Work}\label{sec:related}
\vspace{-0.1cm}
%\cite{guo2017countering,song2017pixeldefend,samangouei2018defense,das2018shield}
%A significant number of adversarial detectors has recently been proposed.
To protect DNN-based image classifiers from adversarial attacks, several detection and monitoring methods have recently been proposed. Similar to our work, these methods aim to identify whether a given input image has been manipulated or not. As it will discussed later in the text, unlike our work, the large majority of existing detectors focuses on \textit{single} images that may have been under \textit{digital} attacks. A summary of such works follows.
%
To detect adversarial images, application of image transformation and processing techniques to input images has been proposed in \cite{tian2018detecting,meng2017magnet,liang2018detecting,liu2021feature,idecode}. VG, mentioned earlier, also lies in this category \cite{kantaros2021real}. %In particular, \cite{tian2018detecting} relies on building a DNN-based detector that takes as input $K\times N$ features, where $K$ is the number of applied transformations (e.g., rotation and translation) and $N$ is the number of logits/classes. Nevertheless, this large input space for the DNN detector may render its training computationally challenging especially for large image domains.
%
%MagNet \cite{meng2017magnet} checks if an input image is adversarial by applying a single image transformation and examining the corresponding softmax output. Specifically, MagNet employs auto-encoders to generate transformed images that are reconstructed from the original ones. Similar to \cite{tian2018detecting}, MagNet requires to train a new DNN-based detector for each new dataset.
%A dataset-agnostic detector is presented in \cite{kantaros2021real} that relies on measuring the KL-divergence between the classifier softmax output of the original and a transformed image. Specifically, it is shown that the same lossy compression method can yield high detection performance against digital attacks across different datasets (MNIST, CIFAR10, and ImageNet) and a brightness transformation can be used to detect various physically attacked traffic signs. Conceptually similar approaches against digital attacks are proposed in \cite{liang2018detecting, liu2021feature}.
%In \cite{liang2018detecting}, an adaptive entropy-based noise reduction method is applied to input images yielding a transformed image.
%In \cite{liu2021feature}, a discrete cosine transformation is used to filter out recessive features. In both works, if the label of the image is preserved after this filtering transformation, then the image is identified as clean; otherwise, it is classified as adversarial.
%
%Similar defenses that rely on image transformations have also been proposed in the image purification domain to defend against \textit{digital} attacks only. For instance, \cite{guo2017countering,das2018shield,samangouei2018defense,yin2020defense} apply compression, bit depth reduction, crop ensemble to remove noise and possible adversarial digital components from images. Purification is applied to \textit{all} images, whether they are adversarial or not, which may compromise the accuracy of the DNN on clean images \cite{zheng2016improving,kou2019enhancing}. %In contrast, VG is orthogonal to these approaches as it focuses on \textit{detection}, and not purification, of both digital and physical attacks, without affecting the accuracy of the target classifier. %A potential approach to combine detection and purification approaches is by applying purification only to images that have been detected to be adversarial.
%
%%%%%%%----__________----
Adversarial detectors that rely on identifying adversarial subspaces in a manifold of clean data have also been proposed \cite{feinman2017detecting,jha2018detecting,crecchi2019detecting,pang2018towards, ma2018characterizing,lee2018simple}.
%In fact, it has been observed that adversarial examples lie on low-probability spaces that are close to, but not on, the clean data manifold. Motivated by this, measures to identify adversarial spaces have been investigated \cite{feinman2017detecting,jha2018detecting,crecchi2019detecting,pang2018towards, ma2018characterizing,lee2018simple}. For instance, %in \cite{feinman2017detecting,jha2018detecting, crecchi2019detecting}, kernel density estimation is proposed that selects thresholds on the likelihood of an image. These defenses have also been employed in \cite{pang2018towards} which, additionally, requires training of the target classifier using a reverse-cross entropy objective function.
%the Mahalanobis distance of a test sample from the training set is employed in \cite{lee2018simple} to detect abnormal examples. This distance is used to compute a confidence score. If the confidence score is above a threshold, then the sample is considered normal; otherwise, it is considered adversarial/OOD. %An alternative to density measures has been proposed in \cite{ma2018characterizing} that considers expansion-based measures of intrinsic dimensionality as a potentially effective method of characterizing adversarial examples.
Related is also the recent work in \cite{rossolini2022increasing} that uses coverage criteria to detect adversarial inputs. %Specifically, to set up the detector proposed in \cite{rossolini2022increasing}, a set of clean images is required. Given such inputs, the intermediate results produced by the various layers of the DNN are recorded and used to produce a representation of the covered activation patterns. This representation is called DNN signature that is used at test time to reason about new images.
%
Defenses that rely on training separate DNN that are capable of identifying adversarial images have been proposed as well \cite{fidel2020explainability,cohen2020detecting,liu2022towards}. %For instance, \cite{fidel2020explainability} proposes a DNN detector trained using the internal layers of the target DNN classifier to discriminate between normal and adversarial inputs. %A conceptually similar defense is proposed in \cite{cohen2020detecting} that relies on training a detector using information provided by the DNN activation layers. Inspired by the previously discussed image transformation-based detectors, \cite{liu2022towards} combines multiple image transformations to generate a K-L-based feature vector that is the input to a DNN-based detector. Then the DNN, returns a positive predictive value $\rho$. If $\rho>0.5$ ,the test image is considered adversarial; otherwise, it is considered benign.
%
%Recently, detectors that operate directly on images, independent from the targeted classifier, have been proposed that rely on steganalysis methods \cite{liu2019detection}. However, the defense in \cite{liu2019detection} is attack-specific, since separate detectors must be designed for each type of attack.
Ensembles of previously discussed detectors and defenses against adversarial attacks is considered in \cite{he2017adversarial}.
% that ensemble of weak defenses is not sufficient to provide strongdefense against adversarial examples.
A more comprehensive summary of existing defenses against adversarial inputs can be found in \cite{deng2021deep,bulusu2020anomalous}.
%\textcolor{red}{[currently editing the following text:]}
%\textcolor{red}{[cite also \cite{zhang2020clipped,hwang2019puvae} in the following paragraph]}
We emphasize that the above defenses have been evaluated only on digital attacks and on single input images.
OOD detection techniques proposed in \cite{cai2020real,yang2022interpretable} have been evaluated against a physical attack \cite{boloor2020attacking}. Reconstruction error by variational auto-encoder on a \textit{single input image} relative to the training data is used as the non-conformity measure (NCM) for OOD detection in~\cite{cai2020real}.~\cite{yang2022interpretable} propose computing prototypes from the training data and using the distance of an input image from these prototypes for OOD detection on a \textit{single input image}.
% In that work, given a \textit{single input image}, multiple examples similar to the input are generated using variational auto-encoders. These new samples are used to compute the nonconformity of the input image relative to the training set. This nonconformity score is used to determine whether the image is OOD or not.
A similar line of work for OOD detection in time-series data based on NCM has been proposed~\cite{codit}. This work uses error in equivariance with respect to temporal transformations learned on the in-distribution data as the non-conformity score for detection. They, however, do not consider the detection of adversarial attacks.
%
%The authors provide experiments on CARLA to demonstrate that the proposed method can be used in real-time while attaining a low false-alarm rate against a physical attack \cite{boloor2020attacking}.
%
Defenses against adversarial patches have also been proposed in \cite{chiang2020certified,xiang2021patchguard,zhang2020clipped,sridhar2022towards}, where robust training methods have been proposed to enhance the resiliency of the DNN classifier against adversarial inputs. %Nevertheless, these methods may sacrifice accuracy on clean images.
%
Unlike these, this paper focuses on \textit{detecting} physical attacks without modifying the structure of the DNN image classifier. In fact, detection frameworks are complementary to robust training methods, as the former essentially aim to monitor the input to DNN models (whether they are trained in a robust fashion or not). Moreover, as discussed earlier, we note that the proposed time-series detector is modular in the sense that VG (step (1)) can be replaced by any other single-image detector such as \cite{boloor2020attacking} or \cite{lee2018simple,liang2018detecting}. %, we additionally show how single-image detectors can be applied to detect adversarial inputs in time-series image data.
%
%To the best of our knowledge, this is the first work that investigates how and if historical image data can be used to enhance detection performance \textcolor{red}{[???]}.
Additionally, we provide the first evaluation of adversarial detectors \cite{lee2018simple,liang2018detecting}, originally proposed for digital attacks and/or OOD data, against state-of-the art robust physical attacks \cite{eykholt2018robust,brown2017adversarial} and we show that under certain conditions their performance can be improved by exploiting historical data. %Preliminary results of this work have been presented in \cite{kantaros2021real}. In this paper, we extend \cite{kantaros2021real} (i) extending the proposed single-image detector to account for time-series image data; (ii) providing a theoretical analysis that demonstrates the benefit of leveraging historical data with respect to detection performance; (iii) providing a more thorough experimental analysis against additional detectors \cite{lee2018simple,liang2018detecting} and robust physical attacks \cite{brown2017adversarial}.
%A dataset-agnostic detector is presented in \cite{kantaros2021real} that relies on measuring the KL-divergence between the classifier softmax output of the original and a transformed image. Specifically, it is shown that the same lossy compression method can yield high detection performance against digital attacks across different datasets (MNIST, CIFAR10, and ImageNet) and a brightness transformation can be used to detect various physically attacked traffic signs. Conceptually similar approaches against digital attacks are proposed in \cite{liang2018detecting, liu2021feature}.
%Therefore, their applicability on more realistic physical attacks and on large-scale image domains is questionable as also discussed in \cite{carlini2017adversarial} and shown in our comparative experiments; see Section \ref{sec:sim}. Finally, we would like to emphasize that existing detectors - see e.g., \cite{meng2017magnet,tian2018detecting,feinman2017detecting,jha2018detecting,pang2018towards,cai2020real} - are dataset-specific as they heavily rely on training sets directly and/or building separate DNN models based on these training sets. Particularly, in these works, a new auto-encoder, a new DNN-based detector need to be built, or new embeddings need to be extracted and stored for each dataset. Therefore, it is unclear how these methods perform when they are deployed in real-world environments for which datasets do not exist. To the contrary, VG does not rely on training sets, or on the training process of the target DNN, or on building separate DNN classifiers. For instance, we show through extensive experiments that VG is \textit{dataset-agnostic} in the sense that the same transformation (JPEG with compression quality $92\%$) yields high detection performance across digitally attacked datasets that differ both in content and in image dimensions.
%An alternative research direction to design defense mechanisms - that do not rely on image transformations - focuses on improving robustness of the target DNN via adversarial/robust training approaches. Specifically, adversarial training methods rely on augmenting the training set with adversarial examples and incorporating an adversarial component as a regularizer in the classification objective; see e.g., \cite{moosavi2016deepfool,kurakin2016adversarial} and the references therein. Additionally, distillation, initially proposed to reduce the size of deep neural networks \cite{hinton2015distilling}, can be used as a mechanism to train roust DNN \cite{papernot2016distillation}. %The effectiveness of this defense against JSMA is validated in \cite{papernot2016distillation}.
%Similar, but more computationally efficient training methods, called label smoothing, have also been proposed in \cite{warde201611}.
%A computationally-efficient robust training method is also proposed in \cite{zantedeschi2017efficient} that relies on Gaussian data augmentation during training and requires the BReLU activation function. Note that adversarial/robust training methods \textit{complement} adversarial detectors such as VG. A recent summary of existing defenses can also be found in \cite{yuan2019adversarial}.
%%%%%%%
%\subsection{Evaluation: Scalability \& Detection Performance}\label{sec:eval}
%We evaluate VG against digital attacks on the MNIST, CIFAR10, and ImageNet datasets and we show that, unlike relevant works, it is very computationally light in terms of runtime and memory requirements, even when it is applied to large-scale datasets, such as ImageNet; therefore, it can be employed in real-time applications that may also involve large-scale image spaces. The latter is also demonstrated in a drive-by experiment with clean and physically attacked traffic signs. Moreover, to evaluate the detection performance of VG against physical attacks, we propose the first dataset, called AdvNet, with clean and physically attacked traffic sign images using the $\text{RP}_2$ attack \cite{eykholt2018robust}. We provide extensive comparisons that show that VG outperforms similar detectors \cite{meng2017magnet,feinman2017detecting,jha2018detecting} both in terms of scalability and detection performance.
%
%Note that the performance of the proposed detector can potentially be improved via, e.g., stability training \cite{zheng2016improving}, that aims to make the DNN models more robust to perturbations, such as JPEG compression, since this may decrease false positive alarms. \textcolor{red}{Here we need to add few sentences about our comparisons}
%(ii) provide the first evaluation against this physical attack which also shows that VG achieves high detection performance and outperforms existing detectors designed specifically for digital attacks.
%\paragraph{Scalability:} In particular, VG attains comparable performance to these defenses on small datasets, such as MNIST and CIFAR10, but is more computationally efficient in terms of memory requirements. Due to its low computational cost at both runtime and design-time, VG is suitable for real-time applications that may involve large-scale image domains. as highlighted by our experiment on ImageNet images. To the contrary, application of \cite{meng2017magnet,feinman2017detecting,jha2018detecting} to ImageNet is computationally challenging. Specifically, the defense in \cite{meng2017magnet} requires the training of additional auto-encoders. In fact, we were unable to train an auto-encoder for ImageNet within two weeks of training. The detector in \cite{feinman2017detecting,jha2018detecting}, on the other hand, requires extracting and storing the last hidden layer output for all training images which is a time-consuming process and may not be possible on all platforms (e.g., lightweight IoT cameras) due to excessive memory requirements. These embeddings are used at runtime to check if an image is adversarial. The same limitation also holds for \cite{pang2018towards} which employs the KDE detector \cite{feinman2017detecting,jha2018detecting} but additionally requires a specific training process for the target classifier. Similarly, \cite{tian2018detecting} requires training a new DNN detector. Additionally, \cite{tian2018detecting} is significantly more computationally expensive than VG, since at runtime it requires the application of $K$ image transformations and the last hidden layer output of the target classifier for each transformed image. %
%\paragraph{Detection Performance:} Also, we provide extensive experiments that show that VG attains high detection performance against both digital attacks evaluated on MNIST, CIFAR10, and ImageNet datasets and on the $\text{RP}_2$ attack evaluated on the proposed AdvNet dataset. The detectors in \cite{feinman2017detecting,jha2018detecting,meng2017magnet} attain similar performance to VG on MNIST and CIFAR 10 but their performance drops significantly in more complex datasets such as ImageNet. \textcolor{red}{[we need to say something about AdvNet and KDE here once experiments are done]}. In fact their performance is comparable to a random detector. Moreover, we would like to emphasize that the defenses proposed in \cite{meng2017magnet,tian2018detecting,feinman2017detecting,jha2018detecting,pang2018towards} are dataset-specific as they heavily rely on training sets directly and/or building separate DNN models based on these training sets. Particularly, in these works, a new auto-encoder and a new DNN-based detector need to be built and new embeddings need to be extracted and stored for each dataset. Therefore, it is unclear how these methods perform when they are deployed in real-world environments for which datasets do not exist. To the contrary, VG does not rely on training sets, or on the training process of the target DNN, or on building separate DNN classifiers. For instance, we show through extensive experiments that VG is \textit{dataset-agnostic} in the sense that the same transformation (JPEG with compression quality $92\%$) yields high detection performance across digitally attacked datasets that differ both in content and in image dimensions.
%Finally, we would like to highlight that several white-box attacks have been proposed to bypass existing defenses that assume that the structure of the defense mechanism is fully known to the attacker \cite{papernot2016transferability,gilmer2018adversarial,carlini2017adversarial,carlini2017magnet}. Designing defenses against white-box attacks, although an important problem in this field, is out of the scope of this paper as their high computational complexity prohibits them from being applied to real-time scenarios that are of particular interest in this work. We would like to highlight again that our goal is to address an equally important issue which is to develop a computationally light defense mechanism that scales to large image domains, for real-time applications, a task that is particularly challenging for existing defenses both at design- and run-time as shown in our experiments. %Nevertheless, we include preliminary results showing that VG is robust to a certain class of white-box attacks by introducing randomness in the image transformations.
%Finally, note that \cite{tian2018detecting,meng2017magnet,feinman2017detecting,jha2018detecting} have only been evaluated on small-scale datasets such as MNIST and CIFAR10 while their evaluation on large-scale datasets, such as ImageNet, is quite challenging since e.g., it requires training new DNN with large input spaces.
%Finally, note that methods to bypass recent defenses in white-box settings have recently been proposed in \cite{carlini2017adversarial,carlini2017magnet,lu2018limitation,roth2019odds}. Correspondingly, we include preliminary results that show that VG is robust to white-box attacks by introducing randomness to the transformations applied to input images.
%in \cite{meng2017magnet,feinman2017detecting,jha2018detecting}
%VG is shown to be robust to white-box attacks discussed in \cite{carlini2017adversarial} \red{[needs to be validated]}. %; see Section\ref{sec:related} for more details.
%\cite{meng2017magnet,feinman2017detecting}
% \vspace{-0.7cm}
\subsection{Contribution}
% \vspace{-0.1cm}
%{\bf{Contributions:}}
The contributions of this paper can be summarized as follows.
\textit{First}, we introduce a new, simple, yet effective and modular, detection framework for physical attacks in time-series image data.
\textit{Second}, unlike the large majority of existing detectors that focus on digital attacks or OOD data, this paper focuses on physical attacks and provides the first evaluation of these detectors against physical attacks. \textit{Third}, we show that the proposed framework is computationally efficient allowing for its employment in real-time applications, where images are collected online. \textit{Fourth}, we provide conditions for VG (or any other employed single-image detector) and the length of the sliding detection window (or equivalently the size of historical data) that need to be satisfied so that the time-series detector $\text{VG}^*$ performs better than VG (or the corresponding employed single-image detector). \textit{Fifth}, we provide extensive comparative experiments against robust physical attacks demonstrating the efficiency of $\text{VG}^*$. %and competitive detectors originally proposed for digital attacks and/or OOD data. %on videos containing clean and physically attacked traffic signs using various existing single-image detectors demonstrating the efficiency of the proposed time-series detector.
%
%\red{\textit{First}, we propose a novel detector that works directly on input images and relies on lossy compression algorithms.
%
% \textit{Second}, we show that the proposed detector can be integrated with image purification approaches giving rise to a joint detection-repair defense technique; see also the supplemental material. \textit{Third}, we show that the proposed detector outperforms existing defenses that rely on lossless compression algorithms [ICCV paper] or Principal Componenent Analysis of input images. \textit{Fourth}, we validate the efficiency of the proposed detectors on the MNIST and CIFAR10 datasets. }
%Consider an image $x$ that has dimensions $w\times h$ and non-negative integer pixel values in the range $[0,1\dots,N]$
%\JW{don't pick 255 -- make it $N$ and state some typical values of $N$ are 255, 511, 1023, etc. -- unless all images are always between 0 and 255 }
%that are scaled to be between $0$ and $1$. %\footnote{Typical values for $N$ are e.g., $255$ and $65535$, depending on image bit depth.}
%\IL{Not clear to me. Does this mean that values are real number in [0,1]? If so, how the original non-negative integer values get scaled? Divide by max int value? \green{YK: The original non-negative integer values get scaled by dividing by 255. Then we have pixel values that are real numbers in the range [0,1].}}
%If $x$ is a grayscale image, then it can be viewed as a vector in $\mathbb{R}^n$, where $n=w \times h$. Similarly, in case of RGB images, $x$ can be written as a vector in $\mathbb{R}^n$, where $n=3 \times w \times h$.
%
Consider a classifier $f:\ccalX\rightarrow \ccalC$, where $\ccalX$ is the set of images $x\in\mathbb{R}^{m}$, where $m$ is the number of pixels, and $\ccalC$ is the set of labels. Let $L(x)$ and $f(x)$ denote the true and the predicted label of image $x\in\ccalX$, respectively.
%
%
Consider also an attacker that aims to attack $f$ to cause desired misclassifications. In particular, here we consider physical attacks that aim to place adversarial stickers on the surface of objects of interest (e.g., traffic signs) so that an image $x$ that includes a physically attacked object gets a desired (wrong) label $t(x)$ i.e., $f(x)=t(x)$ and $t(x)\neq L(x)$. In section \ref{sec:RP2}, we review a targeted robust physical attack \cite{eykholt2018robust}. In Section \ref{sec:detectPr}, we present the adversarial detection problem that this paper addresses.
%resulting in images $x$ that may include adversarial objects. is to perturb an image $x\in\ccalX$ by $\delta$ so that the difference between the perturbed and the original image is imperceptible and the perturbed image $x^*=x+\delta$ is missclassified, i.e., $f(x^*)\neq L(x)$.
%
%Consider also an adversary that perturbs an image $x\in\ccalX$ by $\delta$ so that $f(x^*)\neq L(x)$, where $x^*$ is the perturbed image, i.e., $x^*=x+\delta$.
%
%In what follows, we provide a summary of existing digital and and physical adversarial attacks that can generate such perturbations $\delta$.
\vspace{-0.2cm}
\subsection{Robust Physical Perturbations}\label{sec:RP2}
% \vspace{-0.1cm}
A targeted robust physical attack is presented in \cite{eykholt2018robust} to generate robust visual adversarial perturbations under different physical conditions. The first step requires to solve the following optimization problem that generates adversarial (digital) noise $\delta$: $\underset{\delta}{\text{min}}~\lambda||M_x \delta||_p+NPS+\mathbb{E}_{x_i\sim X^V}J(f(x_i+T_i(M_x\delta)),y^*),$
%\begin{align}\label{eq:physical}
%\underset{\delta}{\text{min}}~\lambda||M_x \delta||_p+NPS+\mathbb{E}_{x_i\sim X^V}J(f(x_i+T_i(M_x\delta)),y^*),
%\end{align}
where (i) $M_x$ is a mask applied to image $x$ to ensure that the perturbation is applied only to the surface of the object of interest (e.g., on a traffic sign and not in the background); (ii) NPS is a non-printability score to account for fabrication error; (iii) $X^V$ refers to a distribution of images containing an object of interest (e.g., a stop sign) under various environmental conditions captured by digital and physical transformations (resulting in attacks being robust to various environmental conditions) ; (iv) $T_i(\cdot)$ denotes the alignment function that maps transformations on the object to transformations on
the perturbation (e.g., if the object is rotated, the perturbation is rotated as well); and (v) $y^*$ is the target label.
%
Finally, an attacker will print out the optimization result on paper, cut out the perturbation $M_x$, and put it onto the target object. The perturbation generated using the $L_1$ norm along with its physical application is shown in Figure \ref{fig:attackedl1}. Observe in this figure, that the $L_1$ norm generates a sparse attack vector allowing the attacker to physically implement the attack with black/white stickers, as shown in Fig. \ref{fig:X}. %\textcolor{red}{RK: why not 1(b) and 1(c)?}.
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth]{attacked.png}\vspace{-0.65cm}
\caption{Perturbation generated by the $\text{RP}_2$ attack using the authors-provided code under the $\ell_1$ norm so that a stop, yield, and speed limit 35 sign are misclassified by the LISA CNN \cite{eykholt2018robust} as speed limit 35, speed limit 35, and turn right sign, respectively. These images are used as a guide to place adversarial stickers on the surface of traffic signs as in Fig. \ref{fig:X}.}\vspace{-0.4cm}
\label{fig:attackedl1}
\end{figure}
%\vspace{-0.5cm}
\subsection{Detection Problem}\label{sec:detectPr}
\vspace{-0.1cm}
Consider an input image $x_t$, collected at a time instant $t\geq0$, to the DNN image classifier $f$ which contains a single object of interest. Consider also an ordered sequence $X_{0:t-1}=x_0,x_1,\dots,x_{t-1}$ of images $x_n$, $n\in\{0,1,\dots,t-1\}$ containing the same object of interest as $x_t$ does. For instance, consider an autonomous car approaching a stop sign. In this case, $x_t$ is the current input image - containing the stop sign - to the DNN-based perception system $f$ while $X_{0:t-1}$ denotes a time-ordered sequence of observations/images of the same stop sign taken as the car was approaching it. Our goal in this paper is to build a detection method that determines if $x_t$ is an adversarial image, i.e., if $x_t$ contains an object that has been physically attacked, by leveraging the available historical data $X_{0:t-1}$. We note that in this paper we assume the existence of a perfect object tracking algorithm that is capable of creating an ordered sequence $X_{0:t}$ containing the same object. This assumption can be relaxed by using e.g., recent data association algorithms \cite{bowman2017probabilistic}, which however, is out of the scope of this paper. %Also, for simplicity, we assume that $X_{0:t}$ contain a single object of interest.
The problem that this paper addresses can be summarized as follows:
%As discussed in Section \ref{sec:introduction}, several detectors have been proposed for this purpose.
%In other words, we want to build a detector $f_d:\ccalX\rightarrow\{\text{adversarial, clean}\}$ that classifies input images to $f$ as adversarial or clean.
%\vspace{-0.2cm}
\begin{pr}\label{pr1}
Given (i) a DNN $f:\ccalX\rightarrow\ccalC$ that is subject to physical attacks discussed in Section \ref{sec:RP2}; (ii) an input image $x_t\in\ccalX$ under investigation containing an object of interest; and (iii) an ordered sequence of past observations/images $X_{0:t-1}$ containing the same object of interest as $x_t$ does, our goal is to design a detector $f_d:\ccalX_{0:t}\rightarrow\{\text{adversarial, clean}\}$ that classifies the current input image $x_t$ as adversarial or clean given a sequence of past observations $\ccalX_{0:t-1}$.
\end{pr}
%The main assumption that we make throughout this paper is that both the the attacker and the defender have full knowledge of the DNN $f$. However, the attacker is oblivious to the detector/defense mechanism and the defender is not aware of which specific attack (e.g., CW or FGSM) the DNN is subject to.
To solve Problem \ref{pr1}, we present a time-series detector that builds upon VisionGuard (VG), an existing single-image adversarial detector \cite{kantaros2021real}. To this end, first, in Section \ref{sec:vg}, we review VG. Second, in Section \ref{sec:timeSeriesDet}, we propose a time-series detector to address Problem \ref{pr1}. Finally, in Section \ref{sec:Prop}, we demonstrate the effect of using historical data, that exist in time-series data, on detection performance.
\vspace{-0.1cm}
%three adversarial single-image detectors $\hat{f}_d: \ccalX\rightarrow \{\text{adversarial, clean}\}$ that have been proposed to classify single images $x\in\ccalX$ as adversarial or clean. Particularly, we review VisionGuard (VG) \cite{kantaros2021real}, the Mahalanobis-based detector \cite{lee2018simple}, and the entropy-based detector \cite{liang2018detecting}. We note that \cite{lee2018simple,liang2018detecting} have been evaluated only on digital attacks while \cite{kantaros2021real} has been tested against both digital and physical attacks but not in a time-series context.
\subsection{VisionGuard: A Single-Image Adversarial Detector}\label{sec:vg}
% \vspace{-0.1cm}
%Our goal is to build a detector $f_d:\ccalX\rightarrow\{0,1\}$, such that (i) $f_d(x)=0$ if the image $x$ is a legitimate image and (ii) $f_d(x)=1$ if $x\in\ccalX$ is an adversarial input, i.e., if it has been manipulated/perturbed. In what follows, we propose VisionGuard (VG), an attack-agnostic transformation-based detector; see also Algorithm \ref{alg:detect} and Figure \ref{fig:vg}. %; see also Figure \ref{fig:detectors}.
%\begin{figure}[t]
% \centering
% \includegraphics[width=1\linewidth]{vg.png}
% \caption{Graphical illustration of VisionGuard (gray box). VisionGuard classifies an input image under investigation as clean or adversarial using label-invariant transformations and the structure of the DNN under attack \cite{kantaros2021real}.}
% \label{fig:vg}
%\end{figure}
VG is a dataset- and attack-agnostic detector that relies on the observation that adversarial inputs are not robust to certain transformations in the sense that these transformations change the output of the DNN significantly \cite{kantaros2021real}. Typically, such transformations are (i) label-invariant, i.e., the true class of the object of interest should not change under this transformation, and (ii) squeeze out features that may be unnecessary for correct classification such as adversarial components. %Examples of such transformations are zoom-in, zoom-out, brightness, and cropping.
In \cite{kantaros2021real}, it was shown that lossy compression and brightness transformations can alleviate the effect of digital and physical attacks, respectively.
%On the other hand, to detect physical adversarial attacks we employ physical transformations such as rotation and brightness. More details about how the employed transformations affect the detection performance can be found in Section \ref{sec:sim}. %The proposed detector is also summarized in Figure \ref{fig:KL}.
VG comprises four steps to detect whether an input image $x$ is adversarial or not. %; see also Alg. \ref{alg:detect}. %; see also Figure \ref{fig:vg}.
First, the input image $x$ is fed to the classifier $f$ to get the softmax output denoted by $\bbg(x)$. Second, a user-specified transformation $t:\ccalX\rightarrow\ccalX$ is applied to $x$ to get an image $x'=t(x)$. Third, $x'$ is fed to the classifier to get the softmax output denoted by $\bbg(x')$. Fourth, $x$ is classified as adversarial if the softmax outputs $\bbg(x)$ and $\bbg(x')$ are significantly different. Formally, similarity between $\bbg(x)$ and $\bbg(x')$ is measured using the K-L divergence measure, denoted by $D_{\text{KL}}(\bbg(x),\bbg(x')$ and defined as follows: $D_{\text{KL}}(\bbg(x),\bbg(x') = \sum_{c\in\ccalC}\bbg_c(x)\log\frac{\bbg_c(x)}{\bbg_c(x')}$
%\begin{equation}
% D_{\text{KL}}(\bbg(x),\bbg(x') = \sum_{c\in\ccalC}\bbg_c(x)\log\frac{\bbg_c(x)}{\bbg_c(x')}
%\end{equation}
where $\bbg_c(x)$ denotes the $c$-th entry in the softmax output vector $\bbg(x)$; in other words, $\bbg_c(x)$ can be viewed as the probability that the class of image $x$ is $c$. Specifically, if
\begin{equation}\label{eq:Jxx}
J(x,x')=\min(D_{\text{KL}}(\bbg(x),\bbg(x')),D_{\text{KL}}(\bbg(x'),\bbg(x)))
\end{equation}
is greater than a threshold $\tau$, then $x$ is classified as an adversarial input, i.e., $\hat{f}_d(x)=1$ (i.e., $x$ is \text{adversarial}); otherwise, $x$ is classified as a legitimate/clean image, i.e., $\hat{f}_d(x)=\text{clean}$. In \cite{nesti2021detecting}, it is mentioned that performance of VG can be improved if the max, instead of the min, operator is employed in \eqref{eq:Jxx}. Hereafter, to stay consistent with \cite{kantaros2021real}, we use the min operator. The detection threshold is selected using Receiver Operating Characteristics (ROC) graphs. Given an ROC graph computed on a validation set, a threshold that yields high true positive and low false alarm rate is selected. %Note that alternative metrics can also be used in place of K-L divergence such as the Wasserstein and Bregman divergence.
%\vspace{-0.2cm}
\subsection{$\text{VisionGuard}^*$: Detecting Attacks in Time-Series Image Data}\label{sec:timeSeriesDet}
%\vspace{-0.1cm}
%\begin{algorithm}[t]
% \caption{VisionGuard: Single-Image Detector \cite{kantaros2021real}}
% \label{alg:detect}
%\begin{algorithmic}
% \STATE {\bfseries Input:} \{Input image $x$, DNN $f:\mathbb{R}^n\rightarrow\ccalC$, transformation $t$\}
% \STATE {\bfseries Output:} \{$\hat{f}_d(x)=1$ if $x$ is an adversarial input, and $\hat{f}_d(x)=0$ otherwise\}
% %\REPEAT
% \STATE Feed $x$ to the DNN model and get softmax output $\bf{g}(x)$\label{det:gx}
% \STATE Apply transformation $t$ to $x$ and get image $x'=t(x)$.
% \STATE Feed $x'$ to the DNN model and get softmax output $\bf{g}(x')$
% \STATE Compute $J(x,x')=\min(D_{\text{KL}}(\bbg(x),\bbg(x')),D_{\text{KL}}(\bbg(x'),\bbg(x)))$\;\label{det:KL}
% \STATE Initialize $\hat{f}_d(x)=0$
% \IF{$J(x,x')\geq \tau$}
% \STATE $\hat{f}_d(x)=1$
% \ENDIF
% %\UNTIL{$noChange$ is $true$}
%\end{algorithmic}
%\end{algorithm}
%In this section, building upon VG, we present a new modular detection framework, called $\text{VisionGuard}^*$ ($\text{VG}^*$), in time series image data.
%single-image detectors as the ones presented in Section \ref{sec:ImageDetector}.
In what follows, we build a new modular time-series detector, called $\text{VisionGuard}^*$ ($\text{VG}^*$), that classifies an image $x_t$ as clean or adversarial, given a sequence $\ccalX_{0:t-1}$ of past images/observations. As mentioned in Section \ref{sec:detectPr}, hereafter, we assume that all images in the sequence $\ccalX_{0:t}$ contain the same object of interest.
%We focus on images that have a single object.
%
%Let $f_d$ be any adversarial detector over single images; see Section \ref{sec:ImageDetector}. Also, l
Particularly, let $X_{0:t-1}=x_0,x_1,\dots,x_{t-1}$ be an ordered sequence of $t-1$ past observations/images $x_n$, $n\in\{0,1,\dots,t-1\}$ and $x_t$ denote the current input image. Also, let $T\geq0$ denote the length of a sliding detection window capturing how many past images from $X_{0:t}$ can be used. To determine whether $x_t$ is an adversarial image, we examine what is the output of $\hat{f}_d$ on the last $T$ images (including $x_t$). Then a majority-vote mechanism is employed to reason about $x_t$; see Alg. \ref{alg:det2}. Formally, the proposed detector is constructed as follows.
%
%For simplicity, let $f_d(x)=1$ and $f_d(x)=0$ denote that the image $x$ is classified as adversarial and clean, respectively, by the detector $f_d$.
First, we compute the average output of $\hat{f}_d$ over the past $T$ images as follows:\footnote{Note that \eqref{eq:majVote} can be written more generally as the weighted average of the detector outputs where e.g., a larger weight is assigned to more important frames. For instance, in an autonomous driving setup, a larger weight to collected images as the car approaches the object of interest (e.g., a traffic sign).}
\begin{equation}\label{eq:majVote}
s(x_t) =
\begin{cases}
\frac{\sum_{n=t-T}^t \hat{f}_d(x_n)}{T} & \text{if}~ t\geq T\\
\frac{\sum_{n=0}^t \hat{f}_d(x_n)}{T} & \text{otherwise}
\end{cases}
\end{equation}
%\begin{figure}[t]
% \centering
% \vspace{-0.2cm}
% \includegraphics[width=1\linewidth]{ICCPS2021__VisionGuard/figures/TimeSeries.png}
% \caption{Graphical illustration of the proposed time-series detector ($\text{VG}^*$). The y-axis shows the output of a single-image detector $f_d(x)$ over a sequence of images $x$ that lie within a sliding detection window of length $T$ and contain an object of interest (e.g., a stop sign). In the considered example, a sequence of stop sign images is provided and the stop sign is classified as adversarial by the time-series detector $f_d^*$.}
% \vspace{-0.2cm}
% \label{fig:timeSeries}
%\end{figure}
%Then we define the following average over the past:
%If the number $t$ of past observations is greater than the length of the sliding window then
Observe that if $s(x_t)\geq 0.5$, then this means that the majority of the past $T$ images, including the current image $x_t$, are classified as adversarial. Otherwise, the majority is classified as clean. Thus, given $s(x_t)$, we define a time-series detector $f_d$ that complies with majority vote, i.e.,
\begin{equation}\label{eq:VGstar}
f_d(x_t|X_{t-T:t-1}) =
\begin{cases}
1 ~(\text{i.e., adversarial}) & \text{if}~ s(x_t)\geq 0.5\\
0 ~(\text{i.e., clean}) & \text{otherwise}
\end{cases}
\end{equation}
\begin{rem}[Variants of the Time-Series Detector]
We note that $\text{VG}^*$ can be extended in various ways. First, any single image detector can be used in place of VG to reason about a single image; see Section \ref{sec:sim1}. Second, to reason about a single image frame $x_n$, majority-vote over the output of an ensemble of single-image detectors can be applied as opposed to simply relying to the VG output. Alternatively, an ensemble of DNN image classifiers can be used (as opposed to a single one considered in this paper) as in \cite{pang2019improving}. Once a single image $x_n$ has been classified as adversarial or clean using any of the above ways, majority vote can be applied as in \eqref{eq:majVote}-\eqref{eq:VGstar}.
%\textcolor{red}{[mention that other single image detectors and weighted majority vote methods can be used as well. Discuss also other variants of this framework (e.g., majority vote over multiple detectors)]}
\end{rem}
%Then we define the following average over the past:
%If the number $t$ of past observations is greater than the length of the sliding window then
\begin{algorithm}[t]
\caption{$\text{VisionGuard}^*$: Time-Series Detector}
\label{alg:det2}
\begin{algorithmic}
\STATE {\bfseries Input:} \{Length $T$ of detection window; Sequence of images $X_{t-T:t}$; Single-Image detector $\hat{f}_d$\}
\STATE {\bfseries Output:} \{$f_d(x_t|X_{t-T:t-1})=1$ if $x_t$ is adversarial, and $f_d(x_t|X_{t-T:t-1})=0$ otherwise\}
%\REPEAT
\STATE Feed images $x_n$ to $\hat{f}_d$ for all $n\in\{t-T, t-T+1, \dots, t\}$ \label{det2:fd}
\STATE Compute $s(x_t)$ as in \eqref{eq:majVote} \label{det2:maj}
\STATE Initialize $f_d(x_t|X_{t-T:t-1})=0$
\IF{$s(x_t)\geq 0.5$}
\STATE $f_d(x_t|X_{t-T:t-1})=1$
\ENDIF
%\UNTIL{$noChange$ is $true$}
\end{algorithmic}
\end{algorithm}
%\vspace{-0.2cm}
\subsection{Enhancing Detection Performance via Historical Data}\label{sec:Prop}
\vspace{-0.1cm}
%[NEEDS SOME EXTRA WORK]
In this section, we provide conditions for the single-image detector $\hat{f}_d$ and the length $T$ of the sliding detection window that need to be satisfied so that the proposed time-series detector $f_d$ achieves better detection performance than $\hat{f}_d$. These conditions are formally stated in the following proposition; see also Fig. \ref{fig:boundT}. %; see Proposition \ref{prop:betterPer}. Additionally, we provide a lower bound on the length of the sliding detection $T$ so that $f_d$ achieves better performance than $\hat{f}_d$. %Specifically, we want to show that the probability of correct classification/output for $f_d$ is smaller than the corresponding probability for $\hat{f}_d$.
\begin{prop}\label{prop:betterPerf}
Let $x_t$ denote an image under investigation and $X_{0:t-1}$ be a time-ordered sequence of images. Assume that $x_t$ and $X_{0:t-1}$ contain the same object of interest. Also let $\hat{f}_d:\ccalX\rightarrow\{0,1\}$ denote a single-image detector with probability $\hat{p}$ of correct classification/output. Then if (i) the outputs of $\hat{f}_d$ over all images in $X_{0:t-1}$ and $x_t$ are independent; (ii) $\hat{p}>0.5$; and (iii) the number $L$ of available past images in $X_{0:t-1}$ to reason about $x_t$
%length $T$ of the sliding detection window
satisfies the following condition:\footnote{Notice that by construction of $f_d$, we have that $L$ is defined as follows: $L=\min\{t, T\}$, i.e., $L=T$ if $t\geq T$ and $L=t$ otherwise.}
\begin{align}\label{eq:boundT}
L> \frac{2[\ln(2)-\ln(1-\hat{p})]}{(2\hat{p}-1)^2},
\end{align}
then, it holds that $p>\hat{p}$, where $p$ denotes the probability of correct classification of the time-series detector $f_d$. %If $T>0$, $\hat{p}>0$, and the outputs of $\hat{f}_d$ over all images in $X_{0:t-1}$ and $x_t$ are independent, then probability of correct classification of the time-series detector $f_d$ denoted by $p$ satisfies $p>\hat{p}$.
\end{prop}
\begin{proof}
This result is shown by applying Hoeffding's inequality.
%
Let $X_{t-T:t}$ be the ordered sub-sequence of images that lie within the sliding detection window of $f_d$. %For simplicity of discussion and notation, hereafter we assume that $t-T\geq 0$, i.e., $L=T$. Thus, in what follows we use interchangeably the terms `length $T$ of window' and number $L$ of available past images. If $t-T< 0$, the proof follows exactly the same logic (the only difference is that in this case $L=t$).
%
We define random variables $Y_n$ that are $1$, if $\hat{f}_d$ classifies the corresponding image $x_n$ correctly, and $0$ otherwise, for all $n\in\{t-T,\dots,t\}$.
\begin{figure}[t]
\centering
\includegraphics[width=0.6 \linewidth]{boundL.png}
\vspace{-0.3cm}
\caption{Graphical illustration of Proposition \ref{prop:betterPerf}. %The x-axis shows the probability of a single-image detector $\hat{f}_d$ and y-axis shows a lower bound on the length of the sliding detection window.
The curve corresponds to the lower bound \eqref{eq:boundT}. Observe that as $\hat{p}\to 1$, the lower bound for the length $L$ that needs to be satisfied so that the time series detector $f_d$ performs better than the single-image detector $\hat{f}_d$ decreases. For example, if $\hat{p}=0.6$ then $L> 40.24$, while if $\hat{p}=0.9$ then $L> 4.69$. }\vspace{-0.5cm}
\label{fig:boundT}
\vspace{-0.2cm}
\end{figure}
%Given any image $x$, let $p$ be the probability of correct classification for the detector $\hat{f}_d$; which can be learned from data. Also, given a sequence $X_{t-T:t}$ of images that lie within a detection window, let $Y_n$ be a random variable that is $1$ is $\hat{f}_d$ classifies an image $x_n$ correctly, and $0$ otherwise, for all $n\in\{t-T,\dots,t\}$.
%
%Next we compute the probability $p$ that $f_d$ generates the correct classification for $x_t$. Let $c(x_t)$ denote the correct label of $x_t$, i.e., $c(x_t)\in\{\text{clean}, \text{adversarial}\}$.
Next, we define $Y_{x_t}=\sum_{n=t-L}^t Y_n$, i.e., $Y_{x_t}$ is a random variable counting the number of correct classifications of $\hat{f}_d$ over the sub-sequence of images $X_{t-L:t}=x_{t-L}, x_{t-L+1},\dots,x_t$. By definition, $Y_{x_t}$ follows a binomial distribution with mean $T\hat{p}$. By construction of $f_d$ the probability that $f_d$ misclassifies $x_t$, satisfies: $\mathbb{P}(f_d(x_t)\neq c(x_t))=1-p=\mathbb{P}(Y_{x_t}<\frac{L}{2})$,
%\begin{align}\label{eq1}
% \mathbb{P}(f_d(x_t)\neq c(x_t))=1-p=\mathbb{P}(Y_{x_t}<\frac{T}{2})
%\end{align}
where $c(x_t)$ denotes the correct `adversarial' label of $x_t$, i.e., $c(x_t)\in\{\text{clean}, \text{adversarial}\}$. Thus, we have that:
$\mathbb{P}\left(f_d(x_t)\neq c(x_t)\right)=\mathbb{P}\left(Y_{x_t}<\frac{L}{2}\right)\leq \mathbb{P}\left(Y_{x_t}-L\hat{p}<\frac{L}{2}-L\hat{p}\right)\leq \mathbb{P}\left(|Y_{x_t}-L\hat{p}|>\frac{L(2\hat{p}-1)}{2}\right)= \mathbb{P}\left(|Y_{x_t}-\mathbb{E}(Y_{x_t})|>\frac{L(2\hat{p}-1)}{2}\right)\leq 2e^{-\frac{L(2\hat{p}-1)^2}{2}}$%\label{eq:upBnd1}$
%\begin{align}\label{eq:UpperBound}
%&\mathbb{P}\left(f_d(x_t)\neq c(x_t)\right)=\mathbb{P}\left(Y_{x_t}<\frac{T}{2}\right)\leq \mathbb{P}\left(Y_{x_t}-T\hat{p}<\frac{T}{2}-T\hat{p}\right)\nonumber\\
%&= \mathbb{P}\left((Y_{x_t}-T\hat{p}<\frac{T(1-2\hat{p})}{2}\right)\leq \mathbb{P}\left(|Y_{x_t}-T\hat{p}|>-\frac{T(1-2\hat{p})}{2}\right)\nonumber\\
%&\leq \mathbb{P}\left(|Y_{x_t}-T\hat{p}|>\frac{T(2\hat{p}-1)}{2}\right)= \mathbb{P}\left(|Y_{x_t}-\mathbb{E}(Y_{x_t})|>\frac{T(2\hat{p}-1)}{2}\right)\nonumber\\&
%\leq 2e^{-\frac{T(2\hat{p}-1)^2}{2}}%\label{eq:upBnd1}
%\end{align}
%\begin{align}\label{eq:UpperBound}
%&\mathbb{P}\left(f_d(x_t)\neq c(x_t)\right)=\mathbb{P}\left(Y_{x_t}<\frac{L}{2}\right)\nonumber\\
%&\leq \mathbb{P}\left(Y_{x_t}-L\hat{p}<\frac{L}{2}-L\hat{p}\right)\leq \mathbb{P}\left(|Y_{x_t}-L\hat{p}|>\frac{L(2\hat{p}-1)}{2}\right)\nonumber\\
%&= \mathbb{P}\left(|Y_{x_t}-\mathbb{E}(Y_{x_t})|>\frac{L(2\hat{p}-1)}{2}\right)\leq 2e^{-\frac{L(2\hat{p}-1)^2}{2}}%\label{eq:upBnd1}
%\end{align}
where the last inequality is due to Hoeffding's inequality. Note that Hoeffding's inequality can be applied if (a) $Y_n$ are independent random variables, which holds by assumption (i) and (b) $\frac{L(2\hat{p}-1)}{2}>0$, which holds since, by assumption (ii), we have $\hat{p}>0.5$. Observe that the derived upper bound (i.e., $2e^{-\frac{L(2\hat{p}-1)^2}{2}}$) goes to $0$ as $L$ goes to infinity. Next, using this upper bound, we show that if $L$ satisfies \eqref{eq:boundT} then the detection performance of $f_d$ is greater than the performance of $\hat{f}_d$, i.e., $p>\hat{p}$ or, equivalently, $1-p<1-\hat{p}$. In math, we show that the following inequality
\begin{equation}\label{eq:show1}
\mathbb{P}(f_d(x_t)\neq c(x_t))-\mathbb{P}(\hat{f}_d(x_t)\neq c(x_t)<0,
\end{equation}
where $\mathbb{P}(f_d(x_t)\neq c(x_t))=1-\hat{p}$, holds if \eqref{eq:boundT} is true. %In what follows we compute a range of values for $T$ for which \eqref{eq:show1} holds.
%$\mathbb{P}(f_d(x_t)\neq c(x_t))-\mathbb{P}(f_d(x_t)\neq c(x_t)<0$.
An over-approximate range of values for $L$ so that \eqref{eq:show1} is satisfied can be found by using the above derived upper bound. Specifically, in what follows, we compute values for $L$ so that $2e^{-\frac{T(2\hat{p}-1)^2}{2}}-(1-\hat{p})<0$.
%\begin{equation}\label{eq:show2}
% 2e^{-\frac{T(2\hat{p}-1)^2}{2}}-(1-\hat{p})<0
%\end{equation}
Solving this inequality for $L$ yields \eqref{eq:boundT}.\footnote{Note that this requires to assume that $1-\hat{p}>0$, i.e., $\hat{p}<1$ so that $\ln(1-\hat{p})$ is well defined. Nevertheless, if $\hat{p}=1$, then it trivially holds that any length $L\geq0$ can be used to maintain the perfect performance of the single-image detector.}
%\begin{align}\label{eq:Tbound}
% T&> \frac{2[\ln(2)-\ln(1-\hat{p})]}{(2\hat{p}-1)^2}.
%\end{align}
Note that this lower bound is positive since $\ln(2)\approx0.69>0$ while $-\ln(1-p)>0$ due to $1-\hat{p}<1$. Consequently, if the outputs of the single image detector $\hat{f}_d$ are independent, $\hat{p}>0.5$, and $L$ satisfies \eqref{eq:boundT}, then $p>\hat{p}$ completing the proof.%Thus, the above lower bound is meaningful in the sense that it is strictly greater than zero.
%
%that requires the random variables $Y_n$ to be independent (needs to be discussed a bit more).
%Observe that $\mathbb{P}(f_d(x_t)\neq c(x_t))=1-p\geq 1-2\hat{p}$, if $p>0$, where $1-p$ is the probability of misclassification for $\hat{f}_d$ completing the proof.
%
%Since $Y_{x_t}$ takes only positive values, we have that $\mathbb{P}(Y_{x_t}<\frac{T}{2})=\mathbb{P}(|Y_{x_t}|<\frac{T}{2})$. Thus, \eqref{eq1} can written as follows:
%\begin{align}
%\mathbb{P}(f_d(x_t)\neq c(x_t))&=\mathbb{P}(Y_{x_t}<\frac{T}{2})=\mathbb{P}(|Y_{x_t}|<\frac{T}{2})\\
%&=1-\mathbb{P}(|Y_{x_t}|\geq\frac{T}{2})\\
%&\geq 1-\frac{E(|Y_{x_t}|)}{T/2} =1-2\hat{p},
%\end{align}
%where the last inequality is due to Markov's inequality. Note that Markov's inequality can be applied since $Y_n$ are independent random variables and $T>0$. %that requires the random variables $Y_n$ to be independent (needs to be discussed a bit more).
%Observe that $\mathbb{P}(f_d(x_t)\neq c(x_t))=1-p\geq 1-2\hat{p}$, if $p>0$, where $1-p$ is the probability of misclassification for $\hat{f}_d$ completing the proof.
\end{proof}
%\vspace{-0.3cm}
\begin{rem}
We note that if the single-image detector satisfies the conditions (i)-(iii) of Prop. \ref{prop:betterPerf}, then as $L\to\infty$, the probability of misclassification for the time-series detector goes to $0$. %; see the upper bound in \eqref{eq:UpperBound}.
Also, note that Prop. \ref{prop:betterPerf}, implies that the length $T$ of the sliding window should be designed so that it is larger than the lower bound captured in \eqref{eq:boundT}. %Also, assumption (i) is reasonable due to the training process of single-image detectors [\textcolor{red}{[needs to be re-phrased]}].
%as their output on a given image does not depend on their output on another image
\end{rem}
%\textcolor{red}{[Here I need to add a short discussion on the assumptions of this proposition..]}
%[SOMETHING MUST BE WRONG HERE]Next we examine how long the sliding window should be to ensure that the probability of misclassification of the $f_d$ is at least $\delta\in[0,1]$ smaller than the corresponding probability of $\hat{f}_d$. Specifically, we want find values for $T$ so that $\mathbb{P}(f_d(x_t)\neq a(x_t))-(1-p)+\delta\leq 0$.
%\begin{align}
%\mathbb{P}(f_d(x_t)\neq %c(x_t))&=\mathbb{P}(Y_{x_t}<\frac{T}{2})\\
%&\leq \mathbb{P}(Y_{x_t}-Tp<\frac{T}{2}-Tp)\\
%&= \mathbb{P}(Y_{x_t}-Tp<\frac{T(1-2p)}{2})\\
%&\leq \mathbb{P}(|Y_{x_t}-Tp|>-\frac{T(1-2p)}{2})\\
%&\leq \mathbb{P}(|Y_{x_t}-Tp|>\frac{T(2p-1)}{2})\\
%&= \mathbb{P}(|Y_{x_t}-\mathbb{E}(Y_{x_t})|>\frac{T(2p-1)}{2})\\
%&\leq2e^{-\frac{T(2p-1)^2}{2}}
%\end{align}
%The last inequality is due to the Hoeffding inequality which requires the random variables $Y_n$ to be independent (needs to be discussed a bit more) and $\frac{T(2p-1)}{2}\geq 0$ i.e., $p\geq 0.5$. Observe that the probability of misclassification for proposed detector goes to $0$ as the length $T$ of the sliding window goes to infinity.\footnote{\textcolor{red}{This is counter intuitive to me. What if $p=0$? The limit still goes to to 0.}}
%Solving $\mathbb{P}(f_d(x_t)\neq a(x_t))-(1-p)+\delta\leq 0$ for $T$ yields
%\begin{align}
% T&\geq \frac{2[\ln(2)-\ln(1-p-\delta)]}{(2p-1)^2}
%\end{align}
%Note that $\ln(2)\approx0.69>0$ while $\ln(1-p-\delta)>0$ since $1-p-\delta<0$. Thus, the above lower bound is meaningful in the sense that it is strictly greater than zero.
%If $p>0.5$, the probability of misclassification for
%\textcolor{red}{[CURRENTLY WORKING ON THIS SECTION]}
In this section, we extensively demonstrate the performance of $\text{VG}^*$ and compare its performance against baseline detectors \cite{lee2018simple,liang2018detecting} summarized in Section \ref{sec:sumDetExp}. Particularly, in Sections
\ref{sec:setupSingle}, we first discuss how these baseline time-series detectors are set up. In Section \ref{sec:evalDet}, we present the data collection process that involves recording videos of clean and adversarial traffic signs.
%compare the detection performance of VG against the single-image image detectors presented in . These experiments consider images of traffic signs with adversarial stickers on their surface generated by the $\text{RP}_2$ attack discussed in Section \ref{sec:RP2}.
%
Then, in Sections \ref{sec:evalDet2}- \ref{sec:evalDet3}, we compare the performance of $\text{VG}^*$ on the above-mentioned videos against the baseline time-series detectors. %In our experiments, we consider recorded videos of clean and $\text{RP}_2$-attacked traffic signs.
Our experiments validate that historical data can enhance performance of single-image detectors, as shown in Proposition \ref{prop:betterPerf}. In summary, our comparative experiments show that the proposed detector is characterized by a more consistent performance than the competitive detectors whose performance changes significantly across attacks, datasets, and DNN models. Finally, we discuss the real-time performance of the proposed detectors. All experiments have been executed on a computer with Intel(R) Xeon(R) Gold 6148 CPU, 2.40GHz. %In the Appendix, we also compare their performance against adversarial patches presented in \cite{brown2017adversarial}.
%on single images as well as videos containing single traffic signs while considering three different single image detectors: VisionGuard \cite{kantaros2021real}, the Mahalanobis detector \cite{lee2018simple}, and the entropy-based detector \cite{liang2018detecting}. These detectors were reviewed in Section \ref{sec:ImageDetector}. In our evaluation, we have used code provided by the authors. In Section \ref{sec:setupSingle}, we present the target DNNs and we discuss how we have configured/trained the single image detectors using AdvNet, a dataset with clean and physically attacked traffic signs \cite{kantaros2021real}. Then in Sections \ref{sec:evalDet} and \ref{sec:realtime}, we demonstrate the detection performance and the computational efficiency of the resulting time-series detectors. All experiments have been executed on a computer with Intel(R) Xeon(R) Gold 6148 CPU, 2.40GHz. %\textcolor{red}{[this section is a bit dry...]}
% $0.0016$, $0.0040$, $0.00057$
%\begin{table}[t]
%\caption{Average Detection runtime (secs) of VisionGuard per dataset}%, MagNet, and KDE
%\centering
%\label{tab:runtimeDet}
%\begin{tabular}{|l|l|l|l|}
%\hline
% & MNIST & CIFAR10 & ImageNet \\ \hline
%VG + JPEG & $0.0016$ & $0.0187$ & $0.08$ \\ \hline
%%MagNet & $0.0040$ & $0.0213$ & DNF ($>2$ weeks) \\ \hline
%%KDE & $0.00057$ & $0.00047$ & ?? \\ \hline
%\end{tabular}
%\end{table}
%4.7 x 10^(-4)
%\begin{table}[]
%\caption{AUC}%Robustness of VisionGuard to Random Noise
%\centering
%\begin{tabular}{|l|l|l|l|}
%\hline
%Dataset & MNIST & CIFAR10 & ImageNet \\ \hline
%VG+JPEG92 & 53.65\% & 55.35\% & 44.62\% \\ \hline
%%MagNet & ?\% & ?\% & ?\% \\ \hline
%%KDE & ?\% & ?\% & ?\% \\ \hline
%\end{tabular}
%\end{table}
%In total, the constructed dataset consists of clean stop signs, clean yield signs, clean speed limit 35 signs, adversarial stop signs for GTSRB-CNN, adversarial stop signs for LISA-CNN, and adversarial yield and speed limit signs for LISA-CNN with approximately $1000$ images with dimensions $32\times32\times 3$ per label for the clean and adversarial set.
%\subsection{AdvNet Dataset}
% \vspace{-0.2cm}
\subsection{Summary of Considered Baseline Detectors}\label{sec:sumDetExp}
% \vspace{-0.1cm}
\textbf{Mahalanobis-based Detector (MD):}
This detector relies on computing a confidence score for each test sample; if this confidence score is above a threshold then the test image is considered abnormal or ouut-of-distribution \cite{lee2018simple}. To define this confidence score, the Mahalanobis distance to the training data set is computed. Specifically, $|\ccalC|$ class-conditional distributions are defined for each layer of the targeted DNN that are assumed to be Gaussian distributions. In math, for each layer $\ell$ of the DNN $f$, the empirical class mean $\mu_{c,\ell}$ and covariance $\Sigma_{\ell}$ of training samples $\{(x_i,y_i)\}_{i=1}^N$, where $y_i$ is the true label of $x_i$, are defined as follows: (i) $\mu_{c,\ell}=\frac{1}{N_c}\sum_{i:y_i=c}f_{\ell}(x_i)$; and $ \Sigma_{\ell}=\frac{1}{N_c}\sum_{c\in\ccalC}\sum_{i:y_i=c}(f_{\ell}(x_i)-\mu_{c,\ell})(f_{\ell}(x_i)-\mu_{c,\ell})^T$,
%\begin{equation}
% \mu_{c,\ell}=\frac{1}{N_c}\sum_{i:y_i=c}f_{\ell}(x_i),
%\end{equation}
%and
%\begin{equation}
% \Sigma_{\ell}=\frac{1}{N_c}\sum_{c\in\ccalC}\sum_{i:y_i=c}(f_{\ell}(x_i)-\mu_{c,\ell})(f_{\ell}(x_i)-\mu_{c,\ell})^T,
%\end{equation}
where $N_c$ is the total number of training samples with label $c$ and $f_{\ell}(x_i)$ refers to the output of the $\ell$-th layer of the DNN $f$. Then, given a new sample $x$, for each layer $\ell$, the following steps are executed. First, the closest class $\hat{c}$ is computed as per the Mahalanobis distance: $\hat{c}=\argmin_c(f_{\ell}(x)-\hat{\mu}_{c,\ell})^T\hat{\Sigma}_{\ell}^{-1}(f_{\ell}(x_i)-\hat{\mu}_{c,\ell})$. Second, a small noise is added to $x$, i.e., $\hat{x}=x-\epsilon\nabla_x(f_{\ell}(x)-\hat{\mu}_{\hat{c},\ell})^T\hat{\Sigma}_{\ell}^{-1}(f_{\ell}(x_i)-\hat{\mu}_{\hat{c},\ell}))$. Third, a confidence score $M_{\ell}$ is computed as: $M_{\ell}=\max_c-(f_{\ell}(\hat{x})-\hat{\mu}_{c,\ell})^T\hat{\Sigma}_{\ell}^{-1}(f_{\ell}(\hat{x})-\hat{\mu}_{c,\ell})$. Once these steps are repeated for all layers $\ell$, the following confidence score is computed which is used for detection: $M=\sum_\ell\alpha_\ell M_\ell$. This detector has been tested against digital attacks and OOD data \cite{lee2018simple}.
%\subsection{Local Intrinsic Dimensionality Detector}
%\textcolor{red}{[Most likely this text will be deleted unless we provide experiments]}
%The Local Intrinsic Dimensionality (LID) detector relies on the observation that the estimated LID of adversarial examples is much higher than that of normal examples while this becomes more pronounced in deeper layers of DNNs \cite{ma2018characterizing}.
%Particularly, given a data sample $x$ and a random variable $R$ describing its distance from other data samples, LID describes the relative rate at which the cumulative distance function $F(r)$ of $R$ increases as the distance $r$ increases from $0$. Since the actual data distribution is unknown, LID cannot be computed; instead, the LID of an example $x$ can be estimated as:
%\begin{equation}\label{eq:lid}
% \text{LID}(x)=-\left(\frac{1}{k}\sum_{i=0}^k\log\frac{r_i(x)}{r_k(x)}\right)^{-1},
%\end{equation}
%where (i) $r_i(x)$ denotes the distance between x and its i-th nearest neighbor within a sample of data points drawn from the data distribution and (ii) $r_k(x)$ is the maximum of the neighbor distances. In practice, the sample set is drawn uniformly from the available training data (omitting $x$ itself). Using this estimate, and two training datasets with normal and adversarial examples, the LID estimate for each image (both clean and adversarial) and each transformational layer in the DNN is computed. Specifically, if $y^i=f^i(x)$ denote the the output of the $i$-th layer of the target DNN given an input image $x$, then LID is computed for $y^i$, for all $i$ using \eqref{eq:lid}. The generated LID estimates are then used as feature values to train classifier that can distinguish clean and adversarial images .
%, and can be estimated using the distances of x to its k nearest neighbors within the sampl
\textbf{Entropy-based Detector (ED):}
%\begin{figure}[t]
% \centering
% \includegraphics[width=0.7\linewidth]{ICCPS2021__VisionGuard/figures/entropicDet.png}
% \caption{Graphical illustration of the entropy-based detector \cite{liang2018detecting}.}
% \label{fig:Entropy}
%\end{figure}
This detector relies on the observation that digital attacks embed an imperceptible amount of noise to images \cite{liang2018detecting}. %Consequently, the information introduced by the digital perturbation should also be less than that of the original image.
Motivated by this observation, the adversarial perturbation is treated as artificial noise and leverages adaptive noise reduction techniques
to reduce its adversarial effect, where aggressiveness of the denoising strategy is selected based on the entropy of the image. %Then, this detector compares the classification results of a given image and its denoised version.
%
%
%The entropy of the image is utilized to measure the complexity of samples. An aggressive noise reduction strategy is adopted for the sample with a low entropy, which may be perturbed heavily by the adversary. For high-entropy samples, a light strategy is employed. Specifically, as illustrated in Fig. 3,
%
In particular, this detector comprises the following steps. %; see also Figure \ref{fig:Entropy}.
First, the entropy of an image $x$ with dimensions $M\times N$ is computed as: $H=-\sum_{i=0}^{255} p_i\log_2 p_i,$
%\begin{equation}
% H=-\sum_{i=0}^{255} p_i\log_2 p_i,
%\end{equation}
where $p_i=f_i/(M\times N)$ and $f_i$ is the frequency of pixel level $i\in[0,1,\dots,255]$. For RGB images, the entropy is computed as the average of entropies over the three channels.
%
Next, the denoising process follows. First, the image gets quantized using a scalar quantization method where all inputs (i.e., pixel values) within a specified interval are mapped to a common value (called codeword), and the inputs in a different interval will be mapped to a different codeword. The number of intervals for uniform quantization is determined using the entropy of the image. Next, the entropy of the quantized sample is computed to determine if spatial smoothing will be applied. Only if the entropy is larger than a threshold, spatial smoothing (e.g., using a box filter) is applied. The image generated by this adaptive denoising process is denoted by $T(x)$.
%First, the image $x$ under investigation is denoised by a detection filter $T$ to generate a filtered sample $T(x)$. An aggressive noise reduction strategy is adopted if the image $x$ has a low entropy, since such images may have been perturbed heavily by the adversary. To the contrary, for high-entropy samples a light noise reduction strategy is employed.
Finally, the original image $x$ and the filtered one $T(x)$ are sent to the target classifier. If the predicted labels for $x$ and $T(x)$ are the same, then $x$ is considered benign/clean. Otherwise, it is identified as adversarial.
\textbf{Time-Series Detectors $\text{MD}^*$ \& $\text{ED}^*$:} Following the same logic as in Section \ref{sec:timeSeriesDet}, we construct the time series detectors $\text{MD}^*$, and $\text{ED}^*$ based on the single-image detectors $\text{MD}$, and $\text{ED}$. Specifically, these detectors are built exactly as $\text{VG}^*$; the only difference is that VG has been replaced by MD and ED.
% \vspace{-0.1cm}
\subsection{Setting Up Time-Series Detectors}\label{sec:setupSingle}
% \vspace{-0.1cm}
In what follows, we present the target DNN models and we discuss how we set up the single image detectors, discussed above, for the $\text{RP}_2$ attack on traffic signs. We have selected AdvNet as a training set to configure all detectors; AdvNet is a recently proposed dataset with clean and physically attacked traffic signs \cite{kantaros2021real}. All detectors are evaluated on a test set constructed by extracting frames of videos with clean and physically attacked traffic signs. More details follow.
\textbf{Target DNN Models:} %\label{sec:targetDNN}
Hereafter, in our evaluation, we consider the LISA-CNN \cite{eykholt2018robust} and the GTSRB-CNN \cite{stallkamp2012man} that can classify traffic signs. Both CNNs take as input $32\times 32$ RGB images while LISA and GTSRB CNN have $17$ and $43$ labels/outputs, respectively. LISA-CNN has
91\% accuracy on the LISA test set and the GTSRB-CNN achieves 95.7\% accuracy on the GTSRB test set.
%\textbf{AdvNet Dataset:} In \cite{kantaros2021real}, the AdvNet dataset is proposed containing $32\times 32$ RGB images with both clean and physically attacked traffic signs (`stop', `speed limit 35', and `yield') that can fool the LISA-CNN and the GTSRB-CNN. The adversarial images were generated by the $\text{RP}_2$ attack using the $\ell_1$ norm (see Section \ref{sec:RP2}). Specifically, given an image with an object of interest (e.g., a stop sign) and a target label (e.g., a speed limit 35 sign), the $\text{RP}_2$ attack was executed to generate the corresponding adversarial image that can fool the target neural network. The resulting (digital) adversarial image was used as a guide to place black and white stickers on the surface of the traffic sign to fool the target perception system. Recall from Section \ref{sec:RP2} that this attack depends on the target neural network since effective physical attacks differ across DNN models; AdvNet contains adversarial images for both LISA-CNN and GTSRB-CNN.
%%Specifically, AdvNet contains images with clean stop signs, clean speed limit 35 signs, clean yield signs, adversarial stop signs for LISA-CNN, adversarial stop signs for GTSRB-CNN, adversarial yield signs for LISA-CNN, and adversarial speed limit signs for LISA-CNN.
%In total, AdvNet consists of $2,645$ clean and and $4,000$ adversarial traffic sign images collected under various environmental conditions that include angle, distance, background, and lightning conditions. In particular, for the LISA CNN, the stop, yield, and speed limit 35 sign were attacked so that they are missclassified as speed limit 35, speed limit 35, and turn right signs, respectively. As for the GTSRB-CNN, stickers are placed on stop signs so that they are classified as speed limit 35 signs. Examples of the generated adversarial images are shown in
%Fig. \ref{fig:X}. Also, the AdvNet dataset can be found on: \url{https://advnet.seas.upenn.edu/}.
\textbf{VG:} As discussed in Section \ref{sec:ImageDetector}, to select a detection threshold $\tau$ for VG, we construct the ROC curve of VG on AdvNet, which captures the trade-off between the true positive (TP) and false positive/alarm (FP) rate for a range of thresholds. Then, based on the ROC curve, we select a threshold that yields a high TP rate and low FP rate. For instance, the ROC curve when the target neural network is the LISA CNN is shown in Figure \ref{fig:ROC1}; the area under this ROC (AUROC) is $89.43\%$. Then, we pick a threshold $\tau=0.43$ which yields a TP rate equal to $80\%$ and a FP rate equal to $19\%$. Following the same process, we select $\tau=0.78$ for the GTSRB CNN. Note that for both CNNs, VG applies brightness transformation with $\text{HSV}=200$.
\begin{figure}[t]
\centering
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_attacks_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.55\linewidth]{ROC_B200_LISA_ALL.pdf}
\vspace{-0.4cm}
\caption{ROC curve (red) corresponding to VG equipped with brightness transformation with $\text{HSV}=200$ on AdvNet for LISA CNN \cite{eykholt2018robust}. The detection threshold $\tau=0.43$ yields $\text{TP rate}=80\%$ and $\text{FP rate}=19\%$ on AdvNet. Each point in the ROC curve corresponds to a detection threshold while the blue line represents the ROC of a random detector.}\vspace{-0.7cm}
\label{fig:ROC1}
\end{figure}
%\textcolor{red}{[next paragraph should be updated]}
\textbf{MD:} To train the Mahalanobis detector, we followed the same process as we did for VG. First, we compute the ROC of this detector using AdvNet. Then, using the ROC curve, we pick a threshold that yields a high TP and a low FP rate. The AUROC of MD for the LISA CNN is $81.2\%$ which is smaller than the AUROC of VG. In fact, given any fixed TP rate, the corresponding FP rate of MD is higher than that of VG. For instance, for $\text{TP rate}=80\%$, MD yields a FP rate that is approximately $45\%$ (while for VG, $\text{TP rate}=19\%$). Thus, prioritizing a low FP rate, we select a threshold which yields $\text{TP rate}=50\%$ and $\text{FP rate}=8\%$. Nevertheless, the choice of the threshold is user- and scenario- specific.
%\textcolor{red}{Ramneet, can you report these numbers and make any edits needed in this paragraph? If we can plot the ROC and put it as a sub-figure in Fig. 8 that'd be nice too.}
\textbf{ED:} This detector requires the selection of a spatial smoothing filter \cite{liang2018detecting}. In \cite{liang2018detecting}, it was shown that $5\times 5$ cross, $7\times 7$ cross, $5\times 5$ diamond, $7\times 7$ diamond, and $5\times 5$ box filters yield satisfactory performance against digital attacks. To pick a filter for physical attacks, we evaluated this detector on AdvNet for various cross, box and diamond filters. The best performance was attained using a $3\times 3$ box filter.
% \vspace{-0.1cm}
\subsection{Data Collection \& Evaluation Metric}\label{sec:evalDet}
% \vspace{-0.1cm}
% \& Comparative Evaluation of the Single-Image Detectors
To evaluate the detection performance of $\text{VG}^*$ against the baseline detectors $\text{MD}^*$ and $\text{ED}^*$, we have recorded videos of clean and physically attacked stop, yield, and speed limit 35 signs. The traffic signs have been attacked in the same way as in AdvNet. The videos have an average duration of $12$ seconds and they were taken during daytime by a 12MP smartphone camera at $30$ frames-per-second. Each video contains a single (clean or attacked) traffic sign. At the beginning of the videos, the distance between the camera and the traffic sign is on average $15$m which gradually decreases. Snapshots from a representative video are provided in Fig. \ref{fig:snapshotsVideo}.
To compare the detectors, we extract the frames from each video (to construct the time-series data $\ccalX_{0:t}$) and we report the accuracy of each detector defined as follows:
\begin{equation}\label{eq:acc1}
\text{Acc}(T)=\frac{\text{number of correctly labeled images $x_t$}}{\text{total number of images}},
\end{equation}
where the images $x_t$ are labeled as `clean' or `adversarial' using the past $T$ frames in the video.
%\begin{equation}\label{eq:acc1}
% \text{Acc}=\frac{\text{number of correctly labeled frames}}{\text{total number of frames}},
%\end{equation}
%where the frames can be labeled as clean `clean' or `adversarial'.
%\textbf{Summary of comparative detection performance:} The results are summarized on the second column (denoted by $T=0$) of Tables \ref{tab:VGcleanLISA}-\ref{tab:EntrGTSRB}. The last column in these tables (DNN Acc) shows the accuracy of the DNN image classifier on the extracted frames. These results show that VG outperforms MD and ED against the $\text{RP}_2$ attacks on LISA CNN. Specifically, the average accuracy across all frames from all videos (regardless of the traffic sign) of VG, MD, and ED on clean traffic sign images for LISA CNN is $88\%$, $76\%$, and $76\%$, respectively. This performance difference becomes more pronounced in adversarial traffic signs. In fact, the corresponding average accuracy of VG, MD, and ED for adversarial traffic sign images for LISA CNN is $93\%$, $66\%$, and $65\%$, respectively. %Notice that these numbers can be viewed as as estimates of the accuracy $\hat{p}$ of the single image detectors VG, MD, ED, respectively, mentioned in Proposition \ref{prop:betterPerf}.
%Similar observations are made for the GTSRB-CNN \textcolor{red}{+++}; see Appendix \ref{appA}.
%\textcolor{red}{[ED performs a bit better for GTSRB CNN though...]}
%Then, we use the proposed time-series detector to classify each frame $x_t$ as clean or adversarial. Note that there can be frames without the traffic sign. Thus, $f_d$ uses the past $T$ frames that contained the sign. To reason about whether a traffic sign exists within a frame or not, we train a YOLO v3 object detector - using clean images from AdvNet - that can detect stop, yield and speed limit 35 signs. All of the following versions of the time-series detector have been evaluated on the same videos.
%\textcolor{red}{[+++]}
\vspace{-0.1cm}
\subsection{Comparative Evaluation on LISA CNN}\label{sec:evalDet2}
\vspace{-0.05cm}
% of the Time-Series Detectors
We evaluate the performance of the time-series detectors $\text{VG}^*$, $\text{MD}^*$, and $\text{ED}^*$ on the recorded videos mentioned in Section \ref{sec:evalDet}. %; snapshots are shown in Fig. \ref{fig:snapshotsVideo}.
We use these time-series detectors to classify each frame $x_t$ as clean or adversarial. Note that there can be frames without the traffic sign. Thus, the detector uses the past $T$ frames that contained the sign. To reason about whether a traffic sign exists within a frame or not, we train a YOLO v3 object detector - using clean images from AdvNet - that can detect stop, yield and speed limit 35 signs. All detectors have been evaluated on the same videos. The results are summarized in Tables \ref{tab:VGcleanLISA}-\ref{tab:EntrAdvLISA} for LISA CNN. %The evaluation for GTSRB CNN is summarized in Appendix \ref{appA}.
%
Particularly, we report the accuracy of classifying correctly, as adversarial or clean, the frames on each video for $T\in\{0,3,10,T_{\text{all}}\}$ as per \eqref{eq:acc1}. Notice that when $T=0$, the time-series detector coincides with the corresponding single-image detector. The accuracy reported on $T=0$ can also be viewed as an approximation of the accuracy, denoted by $\hat{p}$ in Proposition \ref{prop:betterPerf}, of the single-image detector computed using the frames of the corresponding video.
%
\textit{Also, by $T=T_{\text{all}}$, we refer to the case where $T$ is large enough so that the time-series detector considers all past frames in $\ccalX_{0:t-1}$ of the video to reason about $x_t$, for all $t\geq 0$.} %Particularly, $T_{\text{all}}$ is simply selected to be larger than the total number of frames in the video.
\begin{figure}[t]
\centering
\includegraphics[width=0.6\linewidth]{experSnapshots.png}\vspace{-0.2cm}
\caption{Snapshots of a recorded video containing a clean stop sign.}
\label{fig:snapshotsVideo}\vspace{-0.5cm}
\end{figure}
\textbf{$\text{VG}^*$}: In Tables \ref{tab:VGcleanLISA}-\ref{tab:VGadvLISA}, we show the average detection performance of $\text{VG}^*$ across five videos of clean traffic signs considering the LISA CNN.
%
We observed that, in general, if the accuracy of the single-image detector $\hat{f}_d$ on each frame is reasonably high (implying that $\hat{p}>0.5$ as estimated by the $T=0$ columns), then when $T$ is very large, the time-series detector $f_d$ achieves high performance on distinguishing clean (Table \ref{tab:VGcleanLISA}) and adversarial (Table \ref{tab:VGadvLISA}) traffic signs. This is also implied by Proposition \ref{prop:betterPerf}. %We emphasize that Proposition \ref{prop:betterPerf} does not imply that the detection performance should monotonically increase as $T$ increases, as it only considers the case when $T\to\infty$.
%Also, when the accuracy of the single-image detector $\hat{f}_d$ on each frame is low (i.e., $\hat{p}<0.5$), then the time-series detector performs poorly. %achieves very high false alarm rate and low true alarm rate. %\textcolor{red}{RK: I am confused here, in table 2 with adversarial traffic sign, shouldn't the no.s indicate how many adversarial are detected as adversarial. So, high detection instead of low false alarm? Similar confusion for table 1..}
%For instance, %observe the performance of $\text{VG}^*$ in video 6 containing an adversarial yield sign in Table \ref{tab:VGadvLISA}.
%observe in Table \ref{tab:VGadvLISA}, in video 6 of the adversarial yield sign, that the performance when $T=0$ is low ($44\%$). When $T$ is large enough (i.e., $T=T_{\text{all}}$), the performance gets significantly worse ($0.5\%$). Nevertheless, in this scenario where VG performs poorly, the accuracy of the LISA CNN is relatively high ($63\%$). Additionally, observe in Table \ref{tab:VGcleanLISA}, in video 5 of the clean stop sign, that although the performance when $T=0$ is quite high ($82\%$), the accuracy is lower when $T=T_{\text{all}}$ ($67\%$). This is because of the (single-image) VG output on the first $80$ frames of the video. In total this video has $262$ frames. For the first frames, every $8$-$15$ consecutive frames classified (wrongly) as adversarial are followed by $2$-$4$ frames classified (correctly) as clean images. When majority-vote is applied in the first $80$ frames, these few frames that are correctly classified as clean will switch to adversarial resulting in that accuracy drop. We emphasize that this does not contradict Proposition \ref{prop:betterPerf}, since (i) an accurate estimation of $\hat{p}$ and $p$ requires a large number of data and (ii) the condition captured in \eqref{eq:boundT} in Proposition \ref{prop:betterPerf} does not hold for the first $K$ frames, where $K$ is the right hand side in \eqref{eq:boundT}. %accuracy reported on the $T_{\text{all}}$ column does not capture $p$ mentioned in
%the accuracy of the single-image detector (VG) on the first $80$ frames is pretty low.
%
%Also, we note that in this video, the single-image detector classified $214$ frames, out of $262$ frames, as clean. Thus, overall, using the whole image sequence/video, the time-series detector classifies the stop sign (i.e., image $x_{262}$) as adversarial. %Similar observations are made when this time-series detector is applied considering the GTSRB CNN; see Table \ref{tab:VGgtsrb} in Appendix \ref{appA}.
%
%the first $80$ frames out of the $262$ frames of the video, the majority of them ($48$ frames) are classified as adversarial by VG while
%that the greater the $T$, the greater the detection accuracy, i.e., the lower the false alarm rate. The
%
%The same holds for the adversarial signs as well, (see Tables X4-X6) implying that increasing $T$ results in higher true alarm rates.
%that although the object detector is imperfect, the time series detector can still attain high dete
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[t]
\caption{$\text{VG}^*$: LISA CNN And \textit{CLEAN} Traffic Signs}\vspace{-0.3cm}
\label{tab:VGcleanLISA}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Stop & 87.4\% & 89\% & 88.6\% & 90.4\% & 93\% & 93.8\% \\ \hline
Speed35 & 85.4\% & 87.2\% & 87.4\% & 95.8\% & 98.6\% & 93.8\% \\ \hline
Yield & 92.6\% & 95\% & 97\% & 100\% & 100\% & 99.2\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\begin{table}[t]
\caption{$\text{VG}^*$: LISA CNN And \textit{ADVERSARIAL} Traffic Signs}\vspace{-0.3cm}
\label{tab:VGadvLISA}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Stop & 97.3\% & 98.8\% & 99.8\% & 99.8\% & 99.8\% & 1.6\% \\ \hline
Speed35 & 95.8\% & 96.6\% & 97.2\% & 97.8\% & 97.8\% & 0.5\% \\ \hline
Yield & 86.3\% & 88.5\% & 89.8\% & 90.2\% & 83.3\% & 22.3\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\textbf{$\text{MD}^*$}: The average performance of $\text{MD}^*$ across five videos for LISA CNN is summarized in Tables \ref{tab:MahCleanLISA}-\ref{tab:MahAdvLISA}. Observe that this detector performs quite well on distinguishing clean and adversarial stop and speed limit signs but it performs poorly on yield signs. A potential reason for this is that MD assumes that class-conditions distributions are Gaussian distributions across all layers of the target neural network. This assumption was validated for digitally attacked and OOD data in \cite{lee2018simple}. Nevertheless, it may not hold for certain physical attacks, as the one considered here.
%
Notice that as $T$ increases, the performance of $\text{MD}^*$ does not increase for the yield sign. However, this does not contradict Proposition \ref{prop:betterPerf} as the reported accuracy may be inaccurate (as only a few tens of image frames are used). Recall also that Proposition \ref{prop:betterPerf} does not imply that the detection performance should monotonically increase as $T$ increases. Overall, using the whole image sequence/video, the time-series detector classifies the yield sign (on average across all videos) as clean.
%Also, when the accuracy of the single-image detector $\hat{f}_d$ on each frame is low (i.e., $\hat{p}<0.5$), then the time-series detector performs poorly. %achieves very high false alarm rate and low true alarm rate.
%
In total, the detection performance of $\text{MD}^*$ and $\text{VG}^*$ is comparable on stop signs but $\text{VG}^*$ outperforms $\text{MD}^*$ on speed limit/yield signs. %Finally, similar to $\text{VG}^*$, notice that if the detection performance of the single-image detector MD is high, i.e., $\hat{p}>0.5$ (see $T=0$ for an approximate $\hat{p}$), then the performance of $\text{MD}^*$ increases as the length $T$ of the detection window increase, as expected due to Proposition \ref{prop:betterPerf}. Similarly, when $\hat{p}<0.5$ when $T=0$, then the performance of $\text{MD}^*$ drops as $T$ increases; for instance, see videos 1, 4, 5 and 6 in Table \ref{tab:MahAdvLISA} containing an adversarial yield sign. The evaluation on the GTSRB CNN can be found in Appendix \ref{appA}.
\textbf{$\text{ED}^*$}: The average performance of $\text{ED}^*$ across five videos for LISA CNN is summarized in Tables \ref{tab:EntrCleanLISA}-\ref{tab:EntrAdvLISA}. Similar to $\text{MD}^*$, observe that the single-image detector ($T=0$) performs much worse than VG in several videos. The reason is because ED is built using filters aiming to remove digital noise as opposed to e.g., VG that uses brightness transformations to deal with adversarial stickers. As a result, the performance of this detector is satisfactory on clean traffic signs (see Table \ref{tab:EntrCleanLISA}) but not on physically attacked traffic signs (see Table \ref{tab:EntrAdvLISA}). Overall, this detector seems to performs quite well on distinguishing clean and adversarial stop signs but it fails to generalize to speed limit and yield signs. %Similar observations as the ones made for $\text{VG}^*$ and $\text{MD}^*$ are made here as well regarding the relationship between $\hat{p}$ and $T$.
%
%Evaluation on GTSRB-CNN is shown in Appendix \ref{appA}. %Finally, notice that this .
\begin{table}[t]
\caption{$\text{MD}^*$: LISA CNN And \textit{CLEAN} Traffic Signs}\vspace{-0.3cm}
\label{tab:MahCleanLISA}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Stop & 97.4\% & 99.4\% & 99.6\% & 99.8\% & 99.8\% & 93.8\% \\ \hline
Speed35 & 72\% & 80.4\% & 84.2\% & 93.8\% & 99.2\% & 93.8\% \\ \hline
Yield & 61.8\% & 82.4\% & 51.8\% & 54.6\% & 60\% & 99.2 \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\begin{table}[t]
\caption{$\text{MD}^*$: LISA CNN And \textit{ADVERSARIAL} Traffic Signs}\vspace{-0.3cm}
\label{tab:MahAdvLISA}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Stop & 83.7\% & 89\% & 95.2\% & 95.2\% & 98.3\% & 1.6\% \\ \hline
Speed35 & 82.2\% & 89.2\% & 95.8\% & 98.4\% & 99.2\% & 0.5\% \\ \hline
Yield & 34\% & 33\% & 33.4\% & 33.3\% & 33.3\% & 22.3\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\begin{table}[t]
\caption{$\text{ED}^*$: LISA CNN And \textit{CLEAN} Traffic Signs}\vspace{-0.3cm}
\label{tab:EntrCleanLISA}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Stop & 86.8\% & 90.8\% & 91.2\% & 93.8\% & 93.8\% & 93.8\% \\ \hline
Speed35 & 48.2\% & 47\% & 44.2\% & 44.6\% & 40.4\% & 93.8\% \\ \hline
Yield & 95.8\% & 98.6\% & 99.2\% & 99.6\% & 99.6\% & 99.2\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\begin{table}[t]
\caption{$\text{ED}^*$: LISA CNN And \textit{ADVERSARIAL} Traffic Signs}
\vspace{-0.3cm}
\label{tab:EntrAdvLISA}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Stop & 92.7\% & 94.7\% & 97.5\% & 99.7\% & 100\% & 1.6\% \\ \hline
Speed35 & 34\% & 31\% & 33\% & 26.6\% & 11.2\% & 0.5\% \\ \hline
Yield & 68.5\% & 70\% & 73.8 & 72.7 & 77.2 & 22.3\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\textbf{Summary of comparative detection performance:} The comparative performance of the resulting time-series detector for the LISA CNN is summarized in Figure \ref{fig:LISAsum}. Specifically, Figure \ref{fig:cleanL} and \ref{fig:advL} illustrate the average performance of the three time-series detectors across all clean and adversarial traffic signs, respectively, for $T\in\{0,3,10,T_{\text{all}}\}$. Observe that the VG-based time-series detector achieves the best performance on average, since it has been designed to be effective against physical attacks as opposed to \cite{lee2018simple,liang2018detecting} that focus on digital attacks and/or OOD data. Also, notice that among all considered single-image detectors (i.e., $T=0$), VG attains the best performance in terms of both the accuracy metric defined in \eqref{eq:acc1} and the AUROC; see Section \ref{sec:setupSingle}.
%
%The corresponding comparative results for the GTSRB CNN are shown in Appendix \ref{appA} demonstrating that the performance of $\text{MD}^*$ and $\text{ED}^*$ changes significantly as the neural network model or the dataset changes as opposed to $\text{VG}^*$ that achieves a more consistent performance. These comparative results show that existing detectors designed originally for digital and/or OODdata may fail to detect certain physical attacks \cite{kantaros2021real}.
%. Notice that in the GTSRB-CNN case the entropy-based detector achieves the best performance. Nevertheless, as discussed earlier, this detector may fail to generalise across various traffic signs. These results also demonstrate that existing detectors that have been designed for digital and/or OODdata, such as \cite{lee2018simple}, \cite{liang2018detecting}, may fail at detecting physical attacks.
%\begin{figure}[t]
% \centering
% %\includegraphics[width=0.40\linewidth]{accuracyNN_vs_attacks_MNIST_CIFAR10.pdf}}
% %\includegraphics[width=1\linewidth]{figures/CWatt2.png}
% \includegraphics[width=0.7\linewidth]{figures/advPatch.png}
% \vspace{-0.6cm}
% \caption{Example of an adversarial patch; picture borrowed from \cite{brown2017adversarial}. A photo of a banana is classified correctly by VGG 16 as a banana (top). Once the adversarial sticker is placed trained with $y^*=\text{`toaster'}$, the DNN misclassifies the banana as a toaster.}
% \vspace{-0.4cm}
% \label{fig:advPatch}
%\end{figure}
\begin{figure}[t]
\centering
\subfigure{
\label{fig:imgNet1}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.3\linewidth]{figures/pert_img_983.png}}
\subfigure{
\label{fig:imgNet2}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.31\linewidth]{figures/pert_img_997.png}}\vspace{-0.4cm}
\caption{Examples of ImageNet images attacked by \cite{brown2017adversarial} for the target label $y^*=\text{toaster}$.}\vspace{-0.4cm}
\label{fig:AdvPatchImageNet}
\end{figure}
\begin{figure}[t]
\centering
\subfigure[]{
\label{fig:cleanL}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.46\linewidth]{Lisa_clean_v2.png}}
\subfigure[]{
\label{fig:advL}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.46\linewidth]{Lisa_Adv_v2.png}}\vspace{-0.3cm}
%\subfigure[Transformed Clean Images]{
% \label{fig:accTransformed}
% \includegraphics[width=0.32\linewidth]{accuracyNN_transformedTestSet.png}}
\caption{%Figures \ref{fig:acc1MNIST} and \ref{fig:acc2MNIST} show the
Comparison of the constructed time-series detectors against clean (Fig. \ref{fig:cleanL}) and adversarial traffic signs (Fig. \ref{fig:advL}) for the LISA CNN.
}\vspace{-0.2cm}
\label{fig:LISAsum}
\end{figure}
\begin{table}[t]
\caption{$\text{VG}^*$: GTSRB CNN - (clean \& adversarial) stop signs}\vspace{-0.3cm}
\label{tab:VGgtsrb}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Clean Stop & 85.8\% & 89.8\% & 91.4\% & 93\% & 93\% & 82.8\% \\ \hline
Adv Stop & 77.2\% & 81\% & 87.7\% & 88.7\% & 97.2\% & 13.9\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\begin{table}[t]
\caption{$\text{MD}^*$: GTSRB CNN - (clean \& adversarial) stop signs}\vspace{-0.3cm}
\label{tab:MahGTSRB}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Clean Stop & 73.8\% & 78\% & 76.6\% & 82.2\% & 83.6\% & 82.8\% \\ \hline
Adv. Stop & 48.5\% & 47.6\% & 49\% & 50\% & 50\% & 13.9\% \\ \hline
\end{tabular}
\vspace{-0.5cm}
\end{table}
\begin{table}[t]
\caption{$\text{ED}^*$: GTSRB CNN - (clean \& adversarial) stop signs}\vspace{-0.3cm}
\label{tab:EntrGTSRB}
\begin{tabular}{|l|l|l|l|l|l|l|}
\hline
& T=0 & T=3 & T=10 & T=30 & $T_{\text{all}}$ & \begin{tabular}[c]{@{}l@{}}DNN\\ Acc\end{tabular} \\ \hline
Clean Stop & 91.2\% & 94\% & 94.6\% & 97.4\% & 99.6\% & 82.8\% \\ \hline
Adv. Stop & 86\% & 89.8\% & 87.7\% & 93.8\% & 98.3\% & 13.9\% \\ \hline
\end{tabular}
\vspace{-0.3cm}
\end{table}
\vspace{-0.1cm}
\subsection{Comparative Evaluation on GTSRB CNN}\label{sec:evalDet3}
\vspace{-0.1cm}
Next, we compare the performance of $\text{VG}^*$ against the time-series detectors $\text{MD}^*$ and $\text{ED}^*$, when the GTSRB-CNN is considered. %The comparative experiments are run exactly as in Section \ref{sec:evalDet3}.
%
The GTSRB-CNN is trained on the German Traffic Sign Recognition Benchmark. Since we did not have access to German traffic signs for our physical experiments, GTSRB-CNN is evaluated only on videos with US stop signs. The AUROC of VG (with brightness transformation and $\text{HSV}=200$) and MD for GTSRB-CNN on the AdvNet dataset is $97.81\%$ and and $79.9\%$, respectively. To set up the time series detectors, based on these ROCs, we select thresholds for VG and MD that yield $\text{TP rate}=95\%$, $\text{FP rate}=4\%$ and $\text{TP rate}=50\%$, $\text{FP rate}=8\%$. $\text{ED}^*$ is set up using a $3\times 3$ box filter (exactly as in Section \ref{sec:setupSingle}. The comparative performance of the time-series detectors $\text{VG}^*$, $\text{MD}^*$, and $\text{ED}^*$ when the GTSRB-CNN is employed is summarized in Tables \ref{tab:VGgtsrb}-\ref{tab:EntrGTSRB}.
%
As in Section \ref{sec:evalDet2}, these tables report the accuracy, as defined in \ref{eq:acc1}, of the considered time-series detectors for videos containing a clean or adversarial stop sign. The stickers were placed so that the GTSRB-CNN classifies stop signs as speed limit 35 signs as in \cite{kantaros2021real,eykholt2018robust}. Figure \ref{fig:GTSRBsum} visualizes the average performance of $\text{VG}^*$, $\text{MD}^*$, and $\text{ED}^*$ across all clean and adversarial videos reported in Tables \ref{tab:VGgtsrb}-\ref{tab:EntrGTSRB}. Observe first that $\text{ED}^*$ achieves the best (average) performance in distinguishing clean and adversarial stop signs when the GTSRB-CNN is considered. In fact, $\text{ED}^*$ achieved a satisfactory performance on stop signs when the LISA-CNN was considered but it failed to generalize to other traffic signs (see Section \ref{sec:evalDet2}). $\text{VG}^*$ performs comparably to $\text{ED}^*$ for the GTSRB-CNN but it outperforms it when the LISA CNN is considered. $\text{MD}^*$ achieved the worst performance in the GTSRB-CNN setup. Overall, we observe that $\text{VG}^*$ achieves a more consistent and satisfactory detection performance as the DNN model and the dataset change.
% \vspace{-0.5cm}
\begin{figure}[!h]
\centering
\subfigure[]{
\label{fig:cleanG}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.45\linewidth]{GTSRB_clean_v2.png}}
\subfigure[]{
\label{fig:advG}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.45\linewidth]{GTSRB_Adv_v2.png}}
\vspace{-0.2cm}
%\subfigure[Transformed Clean Images]{
% \label{fig:accTransformed}
% \includegraphics[width=0.32\linewidth]{accuracyNN_transformedTestSet.png}}
\caption{%Figures \ref{fig:acc1MNIST} and \ref{fig:acc2MNIST} show the
Comparison of the constructed time-series detectors against clean (Fig. \ref{fig:cleanG}) and adversarial stop signs (Fig. \ref{fig:advG}) for the GTSRB CNN.
}
\vspace{-0.5cm}
\label{fig:GTSRBsum}
\end{figure}
% \vspace{0cm}
\subsection{Real-Time Computational Requirements}\label{sec:realtime}
% \vspace{-0.1cm}
In this section, we evaluate the real-time computational requirements of the three single-image detectors employed above and the resulting time-series detector in terms of required disk-space and execution time. MD requires storing the empirical class mean and covariance for each layer of the DNN that are used at runtime to detect adversarial inputs. Storing this information required $4.4$ MB.
%
%Due to the large disk-space requirements, the KDE-based detectors \cite{feinman2017detecting,jha2018detecting,pang2018towards} may not be applicable to all platforms, especially to lightweight IoT cameras.
In contrast, VG and ED do not have any disk-space requirements, as they do not rely on storing information from the training sets. Instead, they apply image transformations and filtering methods in memory. Also, VG, MD and ED require $0.018$, $0.023$ and $0.049$ secs, on average, to check if a single image is adversarial. Note that these runtimes are implementation specific and they may depend on the image dimensions. For instance, VG can be optimized by computing the softmax output of the original and the transformed image in parallel. %\IL{What is a typical frame rate, e.g. for self-driving car? It would be nice if we can state that VG can run at typical frame rates.}
%Observe that as the dimensions of the input images increase, the runtime of VG increases slightly as well. This is expected as the JPEG compression/decompression time complexity is $O(n)$, where $n$ is the number of pixels \cite{chiou2017complexity}. Observe in the same table that MagNet and VG have similar execution times due to their similar structure but the KDE detector is faster than both as it does not require applying transformation to images. Nevertheless, as discussed earlier, its fast execution time is accompanied with poor performance on large-scale image domains; see Table \ref{tab:imagenet}.
The employed single-image detectors are fast enough for real time applications, such as autonomous cars. For instance, the YOLO neural network \cite{redmon2016you} typically operates in autonomous driving applications at 55 frames per second (FPS) (or at 155 FPS for a small accuracy trade off) i.e., $55$ images are generated per second \cite{yurtsever2020survey}. Thus, if the car camera operates at $55$ FPS, then a new frame/image is generated every $0.018$ seconds. In other words, VG can be used to reason about trustworthiness of each frame while the entropy detector can be used every $0.049/0.018=2.78$ frames. %Nevertheless, this assumes that there is no other delay in the system which is hard to achieve in practice.
%
Finally, applying majority vote is quite inexpensive. For instance, majority vote required on average $7.1\times 10^{-6}$ \text{secs}, $8.1\times 10^{-6}$ \text{secs}, $7.9\times 10^{-6}$ \text{secs}, and $1.1\times 10^{-5}$ \text{secs} for $T=3, 10, 30$ and $T=T_{\text{all}}$, respectively. We note that these runtimes ignore any computational burden incurred by e.g., object tracking algorithms that will generate sequences of images containing the same object of interest.
\vspace{-0.2cm}
\appendix
%\section{Appendix}
\section{Adversarial Patch: Comparative Experiments of Single-Image Detectors}\label{appB}
\vspace{-0.1cm}
%\textcolor{red}{++++}
In what follows, we compare the performance of the detectors VG, MD, and ED against an adversarial patch \cite{brown2017adversarial}.
\textbf{Adversarial Patch:} A targeted adversarial patch is presented in \cite{brown2017adversarial}. Particularly, to fool a DNN classifier, a part of the image is completely replaced with an adversarial patch. This patch is generated as follows.
Given an image $x$, a patch $p$ that may have any shape, patch location $l$, and patch transformations $t$ (e.g. rotations or scaling) a patch application operator $A(p,x,l,t)$ is defined which first applies the transformations $t$ to the patch $p$, and then applies the transformed patch to the image $x$ at location $l$. The trained patch $\hat{p}$ is trained to optimize the objective function: $\hat{p}=\mathbf{E}_{x\sim\mathcal{X},t\sim T, l\sim L}\log P(f(A(p,x,l,t))=y^*)$,
%\begin{equation}
% \hat{p}=\mathbf{E}_{x\sim\mathcal{X},t\sim T, l\sim L}\log P(f(A(p,x,l,t))=y^*)
%\end{equation}
where $\mathcal{X}$ is a training set of images, $T$ is a distribution over transformations of the patch, $L$ is a distribution over locations in the image, and $P(f(A(p,x,l,t))=\hat{y})$ denotes the probability that the classifier will assign to the adversarially patched image $A(p,x,l,t)$ the target label $y^*$. Note that the expectation is over images as well, which encourages the trained patch to work regardless of what is in the background. The trained patch $\hat{p}$ is obtained in \cite{brown2017adversarial} by using a variant of the Expectation over Transformation (EOT) framework presented in \cite{athalye2018synthesizing}. %An example of an adversarial patch is illustrated in Figure \ref{fig:advPatch}.
As shown in \cite{brown2017adversarial}, this attack is effective by either physically placing the sticker in the vicinity of an object of interest or digitally embedding the sticker in the background of given images (see Fig. \ref{fig:AdvPatchImageNet}).
%\textcolor{red}{[Currently editing the rest of this section]}
%\subsubsection{Setup:} In what follows, we present the target DNNs and the setup of VG as well as of the single-image detectors, discussed in Section \ref{sec:sumDetExp}, for the adversarial patch.
\textbf{Dataset:} All detectors are evaluated on the ImageNet (ILSVRC2012) dataset. ImageNet contains $1.4$ million RGB $224\times 224$ images divided in $1.2$ million training images, $50,000$ validation images, $150,000$ testing images, with $1000$ classes. The adversarial patch is trained as in \cite{brown2017adversarial} to fool ResNet 50 with desired label $y^*=\text{toaster}$. Once the patch is generated, we insert it on a random location on a random ImageNet image. Some examples of adversarially attacked ImageNet images are shown in Figure \ref{fig:AdvPatchImageNet}. In our evaluation, we have used the ResNet-50 with $73.6\%$ accuracy on the ImageNet dataset \cite{he2016deep}.
\textbf{Evaluation:} To compare the performance of the detectors, we compute their corresponding ROC. VG equipped with brightness with HSV values equal to 90 and 200 yields AUROC being equal to $90.2\%$ and $93.5\%$, respectively; see Fig. \ref{fig:VG160_patch}. For example, when $HSV=160$ a threshold $\tau=3.6$ yields a TP rate equal to $90\%$ and a FP rate equal to $14.8\%$. %Thus, the precision of VG when $\tau=3.6$ is $0.86$.
%
The AUROC of MD is $84.5\%$ which is lower than that of VG; see Fig. \ref{fig:MDROC_patch}. In fact, by comparing the corresponding ROC curves, we obsere that for a given FP rate, VG achieves a higher TP rate. For example, a threshold $\tau=-0.94$ yields a TP rate equal to $80\%$ and a FP rate equal to $24.9\%$. %Thus, the precision of MD when $\tau=-0.94$ is Z.
%
As in Section \ref{sec:evalDet2}, to configure ED we investigate various spatial filters. The best performance was attained using a $9\times 9$ box mean filter. In this case, ED yields TP and FP rates that are $97\%$ and $32.5\%$, respectively. Notice that although the TP rate is quite high, the corresponding FP rate may be quite high for certain safety critical applications (e.g., autonomous driving). Overall, VG seems to have a more consistent and `robust' performance across both physical attacks considered in the paper and various models and datasets than MD and ED. We note that VG with brightness transformation and $\text{HSV}=200$ attains a high AUROC under the adversarial patch for various DNN models. For instance, for the DNN models inception v3, vgg16, vgg19 the AUROC is $84.92\%$, $96.35\%$, and $95.93\%$.
\begin{figure}[t]
\centering
\subfigure[]{
\label{fig:VG160_patch}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.48\linewidth]{figures/ROCVG_160patch.png}}
\subfigure[]{
\label{fig:MDROC_patch}
%\includegraphics[width=0.40\linewidth]{accuracyNN_vs_CW_MNIST_CIFAR10.pdf}}
\includegraphics[width=0.45\linewidth]{figures/MDROC_BananaPatch.png}}\vspace{-0.3cm}
%\subfigure[Transformed Clean Images]{
% \label{fig:accTransformed}
% \includegraphics[width=0.32\linewidth]{accuracyNN_transformedTestSet.png}}
% \vspace{-0.5cm}
\caption{%Figures \ref{fig:acc1MNIST} and \ref{fig:acc2MNIST} show the
ROC curve (red) corresponding to VG with brightness trasformation and $\text{HSV}=200$ (Fig. \ref{fig:VG160_patch}) and MD (Fig. \ref{fig:MDROC_patch}) on ImageNet and ResNet50 under the adversarial patch \cite{brown2017adversarial}.
}%\vspace{0cm}
\vspace{-0.6cm}
% \label{fig:LISAsum}
\end{figure}
%As expected due to Proposition \ref{prop:betterPerf}, when the accuracy of the single-image detector on a video is greater than $50\%$, then then the performance of the time series detector increases as $T$ increases as well.
%Given a detection window of length $T$, each frame $x_t$ is classified as adversarial or clean using the time-series detector.
%
%First, using clean images from AdvNet, we train a YOLO v3 object detector that can detect stop, yield and speed limit 35 signs.
%Object detector (imperfect)
%Object tracking (perfect)
%Videos (single object)
%\IL{The graph in Fig 9 can be made smaller. Define what are red and blue lines.}
%\begin{figure}[t]
% \centering
% %\includegraphics[width=0.40\linewidth]{accuracyNN_vs_attacks_MNIST_CIFAR10.pdf}}
% \includegraphics[width=1\linewidth]{ICCPS2021__VisionGuard/figures/ROC_B200_LISA_DRIVE.pdf}
% \caption{ROC curve corresponding to VisionGuard equipped with brightness transformation with $\text{HSV}=200$ on stop traffic signs generated during the drive-by test for LISA CNN \cite{eykholt2018robust}; $\text{AUROC}=100\%$. The detection threshold $\tau=0.43$ yields $\text{TP rate}=100\%$ and $\text{FP rate}=0\%$.}
% \label{fig:ROC2}
%\end{figure}
%\IL{In Fig 10, the graph can be made smaller. Define what red and blue lines represent.}
%{\bf{Real-Time Computational Requirements:}}
% T=3: 7.053323694177576e-06
% T=10: 8.14360541266364e-06
% T=30: 7.93096181508657e-06
% T= Inf: 1.0482684986011403e-05
%If e.g., VG requires $0.0179$ secs to reason about trustworthiness of an $32\times32\times3$ image, then it can be used to analyze $1 \text{sec}/0.0179 \text{secs}= 55.86$ frames within a second. $0.0179 \text{secs}* 55 \text{FPS}=4.4$ frames per second. In other words, it can be used to analyze every $12$-th frame assuming that the car camera operates at $55$ FPS. %[to be updated..currently running simulations to receover the numbers..]}
%%% ENTROPY BASED DETECTOR%%
%\textcolor{red}{[THE NEXT TABLES REFER TO THE ENTROPY-BASED DETECTOR]}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{uber_crash}
\BIBentryALTinterwordspacing
{Verge Staff}, ``Uber's fatal self-driving crash: all the news and updates,''
\emph{The Verge}, 2018. [Online]. Available:
\url{http://www.theverge.com/2018/3/28/17174636/uber-self-driving-crash-fatal-arizona-update}
\BIBentrySTDinterwordspacing
\bibitem{karmon2018lavan}
D.~Karmon, D.~Zoran, and Y.~Goldberg, ``Lavan: Localized and visible
adversarial noise,'' \emph{arXiv preprint arXiv:1801.02608}, 2018.
\bibitem{li2019adversarial}
J.~Li, F.~Schmidt, and Z.~Kolter, ``Adversarial camera stickers: A physical
camera-based attack on deep learning systems,'' in \emph{International
Conference on Machine Learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR,
2019, pp. 3896--3904.
\bibitem{eykholt2018robust}
K.~Eykholt, I.~Evtimov, E.~Fernandes, B.~Li, A.~Rahmati, C.~Xiao, A.~Prakash,
T.~Kohno, and D.~Song, ``Robust physical-world attacks on deep learning
visual classification,'' in \emph{Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition}, 2018, pp. 1625--1634.
\bibitem{brown2017adversarial}
T.~B. Brown, D.~Man{\'e}, A.~Roy, M.~Abadi, and J.~Gilmer, ``Adversarial
patch,'' \emph{arXiv preprint arXiv:1712.09665}, 2017.
\bibitem{boloor2020attacking}
A.~Boloor, K.~Garimella, X.~He, C.~Gill, Y.~Vorobeychik, and X.~Zhang,
``Attacking vision-based perception in end-to-end autonomous driving
models,'' \emph{Journal of Systems Architecture}, vol. 110, p. 101766, 2020.
\bibitem{carlini2017adversarial}
N.~Carlini and D.~Wagner, ``Adversarial examples are not easily detected:
Bypassing ten detection methods,'' in \emph{Proceedings of the 10th ACM
Workshop on Artificial Intelligence and Security}.\hskip 1em plus 0.5em minus
0.4em\relax ACM, 2017, pp. 3--14.
\bibitem{papernot2016limitations}
N.~Papernot, P.~McDaniel, S.~Jha, M.~Fredrikson, Z.~B. Celik, and A.~Swami,
``The limitations of deep learning in adversarial settings,'' in \emph{2016
IEEE European Symposium on Security and Privacy (EuroS\&P)}.\hskip 1em plus
0.5em minus 0.4em\relax IEEE, 2016, pp. 372--387.
\bibitem{choi2022argan}
S.-H. Choi, J.-M. Shin, P.~Liu, and Y.-H. Choi, ``Argan: Adversarially robust
generative adversarial networks for deep neural networks against adversarial
examples,'' \emph{IEEE Access}, vol.~10, pp. 33\,602--33\,615, 2022.
\bibitem{bowman2017probabilistic}
S.~L. Bowman, N.~Atanasov, K.~Daniilidis, and G.~J. Pappas, ``Probabilistic
data association for semantic slam,'' in \emph{2017 IEEE international
conference on robotics and automation (ICRA)}, 2017, pp. 1722--1729.
\bibitem{kantaros2021real}
Y.~Kantaros, T.~Carpenter, K.~Sridhar, Y.~Yang, I.~Lee, and J.~Weimer,
``Real-time detectors for digital and physical adversarial inputs to
perception systems,'' in \emph{Proceedings of the ACM/IEEE 12th International
Conference on Cyber-Physical Systems}, 2021, pp. 67--76.
\bibitem{Apollo}
``Apollo's github page,'' \url{https://github.com/ApolloAuto/apollo}, accessed:
2022-06-24.
\bibitem{lee2018simple}
K.~Lee, K.~Lee, H.~Lee, and J.~Shin, ``A simple unified framework for detecting
out-of-distribution samples and adversarial attacks,'' \emph{Advances in
neural information processing systems}, vol.~31, 2018.
\bibitem{liang2018detecting}
B.~Liang, H.~Li, M.~Su, X.~Li, W.~Shi, and X.~Wang, ``Detecting adversarial
image examples in deep neural networks with adaptive noise reduction,''
\emph{IEEE Transactions on Dependable and Secure Computing}, 2018.
\bibitem{tian2018detecting}
S.~Tian, G.~Yang, and Y.~Cai, ``Detecting adversarial examples through image
transformation,'' in \emph{Thirty-Second AAAI Conference on Artificial
Intelligence}, 2018.
\bibitem{meng2017magnet}
D.~Meng and H.~Chen, ``Magnet: a two-pronged defense against adversarial
examples,'' in \emph{Proceedings of the 2017 ACM SIGSAC Conference on
Computer and Communications Security}.\hskip 1em plus 0.5em minus 0.4em\relax
ACM, 2017, pp. 135--147.
\bibitem{liu2021feature}
H.~Liu, B.~Zhao, M.~Ji, Y.~Peng, J.~Guo, and P.~Liu, ``Feature-filter:
Detecting adversarial examples by filtering out recessive features,''
\emph{Applied Soft Computing}, p. 109027, 2022.
\bibitem{idecode}
R.~Kaur, S.~Jha, A.~Roy, S.~Park, E.~Dobriban, O.~Sokolsky, and I.~Lee,
``i{DECOD}e: In-distribution equivariance for conformal out-of-distribution
detection,'' in \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, vol.~36, no.~7, 2022, pp. 7104--7114.
\bibitem{feinman2017detecting}
R.~Feinman, R.~R. Curtin, S.~Shintre, and A.~B. Gardner, ``Detecting
adversarial samples from artifacts,'' \emph{arXiv preprint arXiv:1703.00410},
2017.
\bibitem{jha2018detecting}
S.~Jha, U.~Jang, S.~Jha, and B.~Jalaian, ``Detecting adversarial examples using
data manifolds,'' in \emph{IEEE Military Communications Conference (MILCOM)},
Norfolk, VA, 2018, pp. 547--552.
\bibitem{crecchi2019detecting}
F.~Crecchi, D.~Bacciu, and B.~Biggio, ``Detecting adversarial examples through
nonlinear dimensionality reduction,'' \emph{arXiv preprint arXiv:1904.13094},
2019.
\bibitem{pang2018towards}
T.~Pang, C.~Du, Y.~Dong, and J.~Zhu, ``Towards robust detection of adversarial
examples,'' in \emph{Advances in Neural Information Processing Systems},
2018, pp. 4584--4594.
\bibitem{ma2018characterizing}
X.~Ma, B.~Li, Y.~Wang, S.~M. Erfani, S.~Wijewickrema, G.~Schoenebeck, D.~Song,
M.~E. Houle, and J.~Bailey, ``Characterizing adversarial subspaces using
local intrinsic dimensionality,'' \emph{arXiv preprint arXiv:1801.02613},
2018.
\bibitem{rossolini2022increasing}
G.~Rossolini, A.~Biondi, and G.~Buttazzo, ``Increasing the confidence of deep
neural networks by coverage analysis,'' \emph{IEEE Transactions on Software
Engineering}, 2022.
\bibitem{fidel2020explainability}
G.~Fidel, R.~Bitton, and A.~Shabtai, ``When explainability meets adversarial
learning: Detecting adversarial examples using shap signatures,'' in
\emph{2020 International Joint Conference on Neural Networks (IJCNN)}.\hskip
1em plus 0.5em minus 0.4em\relax IEEE, 2020, pp. 1--8.
\bibitem{cohen2020detecting}
G.~Cohen, G.~Sapiro, and R.~Giryes, ``Detecting adversarial samples using
influence functions and nearest neighbors,'' in \emph{Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2020, pp.
14\,453--14\,462.
\bibitem{liu2022towards}
H.~Liu, B.~Zhao, Y.~Peng, W.~Li, and P.~Liu, ``Towards understanding and
harnessing the effect of image transformation in adversarial detection,''
\emph{arXiv preprint arXiv:2201.01080}, 2022.
\bibitem{he2017adversarial}
W.~He, J.~Wei, X.~Chen, N.~Carlini, and D.~Song, ``Adversarial example defense:
Ensembles of weak defenses are not strong,'' in \emph{11th $\{$USENIX$\}$
workshop on offensive technologies ($\{$WOOT$\}$ 17)}, 2017.
\bibitem{deng2021deep}
Y.~Deng, T.~Zhang, G.~Lou, X.~Zheng, J.~Jin, and Q.-L. Han, ``Deep
learning-based autonomous driving systems: A survey of attacks and
defenses,'' \emph{IEEE Transactions on Industrial Informatics}, 2021.
\bibitem{bulusu2020anomalous}
S.~Bulusu, B.~Kailkhura, B.~Li, P.~K. Varshney, and D.~Song, ``Anomalous
example detection in deep learning: A survey,'' \emph{IEEE Access}, vol.~8,
pp. 132\,330--132\,347, 2020.
\bibitem{cai2020real}
F.~Cai and X.~Koutsoukos, ``Real-time out-of-distribution detection in
learning-enabled cyber-physical systems,'' in \emph{2020 ACM/IEEE 11th
International Conference on Cyber-Physical Systems (ICCPS)}.\hskip 1em plus
0.5em minus 0.4em\relax IEEE, 2020, pp. 174--183.
\bibitem{yang2022interpretable}
Y.~Yang, R.~Kaur, S.~Dutta, and I.~Lee, ``Interpretable detection of
distribution shifts in learning enabled cyber-physical systems,'' in
\emph{2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems
(ICCPS)}.\hskip 1em plus 0.5em minus 0.4em\relax IEEE, 2022, pp. 225--235.
\bibitem{codit}
R.~Kaur, K.~Sridhar, S.~Park, S.~Jha, A.~Roy, O.~Sokolsky, and I.~Lee,
``{COD}i{T}: Conformal out-of-distribution detection in time-series data,''
\emph{arXiv preprint arXiv:2207.11769}, 2022.
\bibitem{chiang2020certified}
P.-y. Chiang, R.~Ni, A.~Abdelkader, C.~Zhu, C.~Studer, and T.~Goldstein,
``Certified defenses for adversarial patches,'' \emph{arXiv preprint
arXiv:2003.06693}, 2020.
\bibitem{xiang2021patchguard}
C.~Xiang, A.~N. Bhagoji, V.~Sehwag, and P.~Mittal, ``$\{$PatchGuard$\}$: A
provably robust defense against adversarial patches via small receptive
fields and masking,'' in \emph{30th USENIX Security Symposium (USENIX
Security 21)}, 2021, pp. 2237--2254.
\bibitem{zhang2020clipped}
Z.~Zhang, B.~Yuan, M.~McCoyd, and D.~Wagner, ``Clipped bagnet: Defending
against sticker attacks with clipped bag-of-features,'' in \emph{2020 IEEE
Security and Privacy Workshops (SPW)}, pp. 55--61.
\bibitem{sridhar2022towards}
K.~Sridhar, S.~Dutta, R.~Kaur, J.~Weimer, O.~Sokolsky, and I.~Lee, ``Towards
alternative techniques for improving adversarial robustness: Analysis of
adversarial training at a spectrum of perturbations,'' \emph{arXiv preprint
arXiv:2206.06496}, 2022.
\bibitem{nesti2021detecting}
F.~Nesti, A.~Biondi, and G.~Buttazzo, ``Detecting adversarial examples by input
transformations, defense perturbations, and voting,'' \emph{arXiv preprint
arXiv:2101.11466}, 2021.
\bibitem{pang2019improving}
T.~Pang, K.~Xu, C.~Du, N.~Chen, and J.~Zhu, ``Improving adversarial robustness
via promoting ensemble diversity,'' in \emph{International Conference on
Machine Learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2019, pp.
4970--4979.
\bibitem{stallkamp2012man}
J.~Stallkamp, M.~Schlipsing, J.~Salmen, and C.~Igel, ``Man vs. computer:
Benchmarking machine learning algorithms for traffic sign recognition,''
\emph{Neural networks}, vol.~32, pp. 323--332, 2012.
\bibitem{redmon2016you}
J.~Redmon, S.~Divvala, R.~Girshick, and A.~Farhadi, ``You only look once:
Unified, real-time object detection,'' in \emph{Proceedings of the IEEE
conference on computer vision and pattern recognition}, 2016, pp. 779--788.
\bibitem{yurtsever2020survey}
E.~Yurtsever, J.~Lambert, A.~Carballo, and K.~Takeda, ``A survey of autonomous
driving: Common practices and emerging technologies,'' \emph{IEEE Access},
vol.~8, pp. 58\,443--58\,469, 2020.
\bibitem{athalye2018synthesizing}
A.~Athalye, L.~Engstrom, A.~Ilyas, and K.~Kwok, ``Synthesizing robust
adversarial examples,'' in \emph{International conference on machine
learning}.\hskip 1em plus 0.5em minus 0.4em\relax PMLR, 2018, pp. 284--293.
\bibitem{he2016deep}
K.~He, X.~Zhang, S.~Ren, and J.~Sun, ``Deep residual learning for image
recognition,'' in \emph{Proceedings of the IEEE conference on computer vision
and pattern recognition}, 2016, pp. 770--778.
\end{thebibliography}