Adversarially Robust Spiking Neural Networks with Sparse Connectivity
Extracted source text · arXiv 2505.15833
\documentclass{article}
\usepackage[final]{cpal_2025}
\usepackage{url}
\usepackage{algorithm}
\usepackage{algcompatible}
\usepackage{subcaption}
\usepackage{amsmath,amssymb,amsfonts,mathtools,bm,dsfont}
\usepackage{booktabs,tabularx,multirow}
\usepackage{makecell}
\usepackage{xcolor}
\usepackage{wrapfig}
% Custom commands
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclarePairedDelimiter{\norm}{\lVert}{\rVert}
\newcommand{\defeq}{\vcentcolon=}
\title{Adversarially Robust Spiking Neural Networks with Sparse Connectivity}
\author{Mathias~Schmolli\textsuperscript{1}, Maximilian~Baronig\textsuperscript{1,2},~Robert~Legenstein\textsuperscript{1},~Ozan~\"{O}zdenizci\textsuperscript{1,3}\\
\textsuperscript{1} Institute of Machine Learning and Neural Computation, Graz University of Technology, Austria\\
\textsuperscript{2} TU Graz - SAL Dependable Embedded Systems Lab, Silicon Austria Labs, Austria\\
\textsuperscript{3} Chair of Cyber-Physical-Systems, Montanuniversität Leoben, Austria\\
\texttt{m.schmolli@alumni.tugraz.at,\{baronig,robert.legenstein,oezdenizci\}@tugraz.at}
}
\begin{document}
\maketitle
\begin{abstract}
Deployment of deep neural networks in resource-constrained embedded systems requires innovative algorithmic solutions to facilitate their energy and memory efficiency.
To further ensure the reliability of these systems against malicious actors, recent works have extensively studied adversarial robustness of existing architectures.
Our work focuses on the intersection of adversarial robustness, memory- and energy-efficiency in neural networks.
We introduce a neural network conversion algorithm designed to produce sparse and adversarially robust spiking neural networks (SNNs) by leveraging the sparse connectivity and weights from a robustly pretrained artificial neural network (ANN).
Our approach combines the energy-efficient architecture of SNNs with a novel conversion algorithm, leading to state-of-the-art performance with enhanced energy and memory efficiency through sparse connectivity and activations.
Our models are shown to achieve up to $100\times$ reduction in the number of weights to be stored in memory, with an estimated $8.6\times$ increase in energy efficiency compared to dense SNNs, while maintaining high performance and robustness against adversarial threats.
\end{abstract}
\section{Introduction}
Resource-efficient deep learning often requires novel algorithmic solutions that can jointly address various considerations of a machine learning system, such as energy efficiency, memory usage, or security and reliability.
Considering the latter, recent works have extensively studied adversarial training based algorithms with traditional artificial neural network (ANN) architectures, to improve robustness against adversarial attacks~\cite{szegedy2013intriguing}.
In the context of adversarially robust optimization, achieving memory efficiency through sparsity in the utilized ANN architecture has been shown to be a challenging problem, since robustness typically improves with increasing network complexity~\cite{madry2017towards,nakkiran2019adversarial}.
Consequently, several studies have explored robustness aware pruning methods for adversarially pretrained dense neural networks~\cite{ye2019adversarial,sehwag2020hydra,zhao2022holistic}.
Following another line of work, there has been increasing interest in achieving adversarial robustness with spiking neural network (SNN) architectures~\cite{kundu2021hire,ding2022snn,ding2024enhancing,liu2024enhancing,ozdenizci2024adversarially}.
The primary motivation of this research problem is the energy-efficiency potential of these models in reliable edge computing applications, since SNNs operate using binary signals that are transmitted over time, which reduces the need for computationally expensive matrix multiplications found in traditional ANNs, and yields energy-efficient event-driven processing capabilities~\cite{roy2019towards,davies2021advancing}.
Along this direction, our work focuses on the intersection of achieving \textit{adversarial robustness} with inherently \textit{energy-efficient} SNN architectures using \textit{sparse connectivity} structures, which has not been previously studied.
Our initial empirical analyses reveal that achieving robustness through standard end-to-end adversarial training with highly sparse feedforward SNNs is computationally challenging and infeasible with spike-based backpropagation through time (BPTT)~\cite{wu2018spatio,wu2019direct}.
Accordingly, we tackle this problem by exploiting stable adversarial training capabilities of ANNs.
We introduce a neural network conversion algorithm that leverages the sparse connectivity structure and weights of a robustly pretrained and pruned ANN, to obtain an adversarially robust and sparse SNN.
Our method can successfully generate feedforward SNNs with layerwise uniform or non-uniform sparse connectivity structures, by utilizing existing adversarial training and robustness aware pruning methods designed for ANNs.
Contributions of this work are summarized as follows:
\begin{itemize}
\item We present a robust and sparse ANN-to-SNN conversion algorithm, combined with a post-conversion sparse SNN finetuning phase. Our method can
inherit robustness properties of an adversarially pretrained ANN to the resulting SNN under strict sparsity constraints.
\item Our algorithm offers the first solution to the problem of achieving sparsity with adversarially robust feedforward SNNs, and scales to larger benchmarks such as TinyImageNet classification, which is not commonly explored in previous SNN robustness studies.
Our models are shown to achieve up to 100$\times$ compression rates, with an estimated 8.6$\times$ increase in energy-efficiency over their dense SNN counterparts.
\item Our approach is agnostic to the adversarial ANN pretraining and robust ANN pruning methods, thus allows the integration of existing and future advancements in adversarially robust training methods in the ANN domain.
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Background}
\subsection{Spiking Neural Networks}
Spiking neural networks perform event-based information processing with binary activations~\cite{maass1997networks}.
Leaky-integrate-and-fire (LIF) neurons in feed-forward SNNs operate according to the dynamics:
\begin{equation}
\boldsymbol{v}^l(t^{-}) = \tau \boldsymbol{v}^l(t-1) + \boldsymbol{W}^l\boldsymbol{o}^{l-1}(t),
\label{eq:lif_potential}
\end{equation}
\begin{equation}
\boldsymbol{o}^l(t) = H(\boldsymbol{v}^l(t^{-}) - V^l_{th}),
\label{eq:lif_spike}
\end{equation}
\begin{equation}
\boldsymbol{v}^l(t) = \boldsymbol{v}^l(t^{-})(1-\boldsymbol{o}^l(t)),
\label{eq:lif_reset}
\end{equation}
where the output spikes from the previous layer $\boldsymbol{o}^{l-1}(t)$ are weighted by the synaptic connectivity matrix $\boldsymbol{W}^l$, the membrane potential leak factor is denoted by $\tau$, the neuron membrane potential before and after firing of a spike at time $t$ is denoted by $\boldsymbol{v}^l(t^{-})$ and $\boldsymbol{v}^l(t)$, $H(.)$ denotes the Heaviside step function, and the neuron firing threshold is denoted by $V^l_{th}$.
For LIF neurons with a hard-reset mechanism, if the neuron emits a spike at time step $t$, the membrane potential is reset to zero via Eq.~\eqref{eq:lif_reset}.
We provide the input $\boldsymbol{o}^0(t)$ to the first layer via \textit{direct coding}, i.e., the input signal intensity (e.g., image pixel) values are applied to the first layer neurons for $T$ simulation timesteps.
Gradient based SNN training can be accomplished via backpropagation through time (BPTT)~\cite{wu2018spatio,wu2019direct,lee2020enabling}, which requires the use of surrogate gradient functions to approximate the discontinuous derivative of the spike function~\cite{neftci2019surrogate}.
Another line of methods obtain SNNs via ANN-to-SNN conversion~\cite{diehl2015fast}.
This is achieved by harnessing pretrained ANN weights and tuning the firing rates of spiking neurons in SNNs to be approximately proportional to the pretrained ANN neuron activations~\cite{rueckauer2017conversion}.
Although earlier conversion methods required high inference latencies~\cite{sengupta2019going}, recent \textit{hybrid conversion} methods alleviated this by using post-conversion BPTT-based finetuning, such that good performance with short inference latencies ($<$10 timesteps) can be achieved~\cite{rathi2020enabling,rathi2021diet}.
\textbf{Energy-Efficiency of SNNs:}
Deep ANNs often require large weight matrix multiplications with floating point multiplication and accumulation (MAC) operations.
In SNNs, however, layer activations are binary, thus only accumulation operations (AC) are performed at the synaptic connections in an event-driven manner, i.e., only when a spike is emitted from pre-synaptic neurons.
Overall energy consumption of MAC operations are significantly higher than AC operations on various types of hardware~\cite{horowitz20141}.
Taking this into account, feedforward SNNs can result in an advantage of energy-efficiency if the spiking activity across the simulation duration remains sufficiently low~\cite{roy2019towards,davies2021advancing}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Adversarial Robustness of Neural Networks}
Neural networks are vulnerable against \textit{adversarial examples} with imperceptible test-time perturbations that lead to incorrect decisions~\cite{szegedy2013intriguing}.
Prominent white-box threats such as fast gradient sign method
(FGSM)~\cite{goodfellow2014explaining} and projected gradient descent (PGD)~\cite{madry2017towards} generate $l_{\infty}$-norm bounded perturbations based on the gradient, where
$\boldsymbol{\tilde x} = \boldsymbol{x} + \epsilon \cdot \text{sign} \left(\nabla_{\boldsymbol{x}} \mathcal{L}(f(\boldsymbol{x};\boldsymbol{\theta}), y))\right)$
indicates the FGSM attack with an $\epsilon>0$ perturbation strength, and PGD is an iterative variant:
$\boldsymbol{\tilde x}_{k+1} = {\textstyle\prod}_{\epsilon}^{\scriptscriptstyle\infty}\left[\boldsymbol{\tilde x}_k + \alpha \cdot \text{sign} \left(\nabla_{\boldsymbol{\tilde x}_k} \mathcal{L}\left(f(\boldsymbol{\tilde x}_k;\boldsymbol{\theta}), y\right)\right) \right],
$
with ${\textstyle\prod}_{\epsilon}^{\scriptscriptstyle\infty}$ being the clipping onto the $l_{\infty}$-norm $\epsilon$-ball around $\boldsymbol{x}$, and $\alpha$ is the perturbation step size.
To counter these attacks, most effective empirical defenses rely on \textit{adversarial training (AT)}, by incorporating adversarial examples into the training process:
\begin{equation}
\underset{\boldsymbol{\theta}}{\min} \; \mathbb{E}_{(\boldsymbol{x},y) \sim \mathcal{D}}\left[ \underset{\boldsymbol{\tilde{x}} \in \mathcal{B}_{\epsilon}^{\scriptscriptstyle\infty}(\boldsymbol{x})}{\max} \mathcal{L}_{\text{robust}}(f(\boldsymbol{\tilde{x}};\boldsymbol{\theta}), y) \right],
\label{eq:robust_training}
\end{equation}
with the inner maximization obtaining adversarial samples in $\mathcal{B}_{\epsilon}^{\scriptscriptstyle\infty}(\boldsymbol{x}) := \{ \boldsymbol{\tilde{x}} : \| \boldsymbol{\tilde{x}} - \boldsymbol{x} \|_{\infty} \leq \epsilon \}$.
Standard AT~\cite{madry2017towards} only uses adversarial samples via
$\mathcal{L}_{\text{robust}}^{\text{AT}} = -\log p(y|\boldsymbol{\tilde{x}};\boldsymbol{\theta})$.
Later methods such as TRADES~\cite{zhang2019theoretically}, MART~\cite{wang2019improving}, or consistency regularization~\cite{tack2022consistency} introduced unique regularization methods to achieve better robustness-accuracy trade-offs.
\textbf{Adversarial Robustness of SNNs:}
Earlier works aimed to improve adversarial robustness of SNNs based on methods designed for ANNs~\cite{sharmin2020inherent,liang2022toward} (e.g., standard AT with BPTT~\cite{sharmin2019comprehensive,marchisio2020spiking}).
Subsequently, various studies examined structural SNN components and their contributions to robustness~\cite{el2021securing,kundu2021hire,xu2022securing}.
Majority of recent methods to improve robustness in SNNs rely on regularized end-to-end AT approaches with BPTT~\cite{ding2022snn,ding2024robust,ding2024enhancing,liu2024enhancing}, which requires computationally heavy optimization procedures for large-scale problems.
Accordingly, from a different perspective, hybrid ANN-to-SNN conversion with adversarially pretrained robust ANNs has achieved state-of-the-art adversarial robustness in SNNs~\cite{ozdenizci2024adversarially}, alleviating the computational burden of end-to-end AT with BPTT.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Sparsity and Adversarial Robustness}
Neural networks often need to meet certain resource constraints in terms of their memory usage on embedded systems.
This has been widely studied in the context of achieving sparsity in models to reduce the number of weights that need to be stored in memory~\cite{han2015learning}.
Accordingly, previous works have also explored sparsity in adversarially trained models via \textit{robust pruning} algorithms~\cite{sehwag2020hydra,zhao2022holistic}.
\textbf{Neural Network Pruning:}
Given a model $f(\boldsymbol{x};\boldsymbol{\theta})$, a pruning algorithm aims to find a global pruning mask $\boldsymbol{m}$ consisting of per-layer binary matrices $\{\boldsymbol{M}^{(l)}\}_{l=1}^L$, which flags a desired global sparsity ratio $\kappa\in[0,1)$ of parameters $\boldsymbol{\theta}$ from the dense model that will be set to zero.
The goal is to obtain a well-performing new model $f(\boldsymbol{x}; \boldsymbol{m} \odot \boldsymbol{\theta'})$ with $\norm{\boldsymbol{m} \odot \boldsymbol{\theta'}}_0\leq(1-\kappa)\cdot n$, where $n$ is the total number of parameters and $\kappa\in[0,1)$ (e.g., $\kappa=0.99$ indicates 99\% sparsity, $\kappa=0$ indicates the dense model).
After determining $\boldsymbol{m}$, remaining parameters are finetuned to recover performance, such that $\boldsymbol{\theta'}$ deviates from the original $\boldsymbol{\theta}$.
During pruning, the significance of each parameter is generally quantified via an importance score vector $\boldsymbol{s}$, which
determines the mask indices $m_j$ based on $s_j$:
\begin{equation}
m_j=\mathds{1}[s_j-\hat{s}_\sigma\geq0],\quad\forall j\in\{1,\ldots,n\},
\label{eq:masking}
\end{equation}
where $\sigma=(1-\kappa)\cdot n$, $\hat{\mathbf{s}}=\text{SortDescending}(\mathbf{s})$, thus $\hat{s}_{\sigma}$ is the $\sigma$-th largest element in $\mathbf{s}$, and $\mathds{1}[.]$ is the indicator function.
A baseline approach is least weight magnitude (LWM) based pruning~\cite{han2015learning}:
\begin{equation}
s_j=|\theta_j|,\quad\forall j\in\{1,\ldots,n\}.
\end{equation}
Here, the pruning rate $\kappa$ is generally applied per-layer to the model such that layerwise uniform sparsity is maintained.
Importantly, LWM can not consider any adversarial robustness criterion while choosing which parameters to prune.
\textbf{Robust Pruning with Learned Importance Scores:}
One of the earlier robustness-aware pruning methods, HYDRA~\cite{sehwag2020hydra}, optimizes $\boldsymbol{s}$ as a learnable parameter under an AT objective.
Specifically, the importance scores are updated with gradient descent (without updating model weights) via:
\begin{equation}
\min_{\mathbf{s}}\mathbb{E}_{(\boldsymbol{x},y)\sim\mathcal{D}}\left[\underset{\boldsymbol{\tilde{x}} \in \mathcal{B}_{\epsilon}^{\scriptscriptstyle\infty}(\boldsymbol{x})}{\max}\mathcal{L}_{\text{robust}}(f(\boldsymbol{\tilde x};\mathbf{m}(\mathbf{s}) \odot \boldsymbol{\theta}),y)\right],
\label{eq:robust_pruning}
\end{equation}
where in each forward pass the current mask $\mathbf{m}(\mathbf{s})$ is obtained via Eq.~\eqref{eq:masking} by maintaining \textit{uniform per-layer sparsity} (i.e., highest ranked $\sigma^{(l)}=(1-\kappa)\cdot n^{(l)}$ scores in the $l$-th layer remain).
All scores $s_j$ are optimized by using a straight-through estimator through the mask during backpropagation~\cite{ramanujan2020}.
The recent holistic adversarially robust pruning (HARP) method~\cite{zhao2022holistic} advanced this idea by simultaneously optimizing how many and which parameters to prune individually in each layer.
This yields \textit{non-uniform per-layer sparsity} ratios that differ from the global sparsity rate $\kappa$.
This is performed by introducing a learnable, continuous-valued representation of layerwise compression rates, such that Eq.~\eqref{eq:robust_pruning} is modified to also optimize layerwise compression quotas $r^{(l)}$ alongside $\boldsymbol{s}$, together with an additional regularizer term that enforces the global sparsity constraint of $\kappa$ based on $r^{(l)}$.
Finally, the resulting mask $\mathbf{m}$ based on the optimized $\mathbf{s}^{\star}$ is used for final pruning and subsequent adversarial finetuning, yielding an adversarially robust and sparse model with parameters $\mathbf{m} \odot \boldsymbol{\theta'}$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Adversarially Robust and Sparse ANN-to-SNN Conversion}
Our approach requires a baseline ANN for conversion, hence we first perform dense ANN adversarial pretraining and pruning.
We then leverage this sparse connectivity and weights via ANN-to-SNN conversion.
Finally, we adversarially finetune the sparse SNN to improve performance.
\subsection{Pretraining Robust and Sparse ANNs}
The initial stage of our method can exploit any existing AT algorithm designed for ANNs, following the objective in Eq.~\eqref{eq:robust_training} for any chosen adversarial perturbation budget $\epsilon$.
After dense ANN adversarial pretraining, we perform robust pruning with learned importance scores via Eq.~\eqref{eq:robust_pruning} to determine which connections to prune from the dense ANN for a given global sparsity ratio $\kappa$.
This is achieved either by preserving \textit{layerwise uniform sparsity}~\cite{sehwag2020hydra}, or by learning layer compression rates (i.e., \textit{layerwise non-uniform sparsity}~\cite{zhao2022holistic}), such that we obtain per-layer sparsity masks $\{\boldsymbol{M}^{(l)}\}_{l=1}^L$.
\subsection{Converting Pretrained ANNs to SNNs}
Our conversion approach relies on the conventional threshold-balancing algorithm~\cite{sengupta2019going,rathi2021diet}, which was also recently explored for robust ANN-to-SNN conversion with \textit{dense} connectivity~\cite{ozdenizci2024adversarially}.
\textbf{Parameter Initialization from Pretrained ANN:}
We use the same network architecture for the SNN using the weights $\{\boldsymbol{W}^{(l)}\}_{l=1}^L$ from the pretrained, sparse and robust ANN as the synaptic connectivity weights, considering the per-layer sparsities imposed via $\{\boldsymbol{M}^{(l)}\}_{l=1}^L$.
We exchange ReLU activations of the ANN with LIF spiking neuron dynamics.
We use threshold-dependent batch-norm operations in the SNN~\cite{zheng2021going} to replace ANN batch-norm layers, where weighted spiking inputs are temporally gathered in $\bar{\boldsymbol{O}}^l = [\boldsymbol{W}^l\boldsymbol{o}^{l-1}(1),\ldots,\boldsymbol{W}^l\boldsymbol{o}^{l-1}(T)]$, and batch-norm is performed on $\bar{\boldsymbol{O}}^l$, i.e., mean and variance statistics are estimated over both mini-batches and time.
We then re-use the pretrained ANN batch-norm affine transformation parameters $\varphi^l$ and $\omega^l$ for the SNN, but now estimate the running average statistics from scratch during SNN finetuning based on the spiking data~\cite{ozdenizci2024adversarially}.
\textbf{Initialization of Firing Thresholds:}
Following weight initialization, we calibrate per-layer firing thresholds of the SNN by using a number of training mini-batch forward passes.
Specifically, we generate inputs to the network via direct coding for a number of calibration timesteps $T_c>T$.
Subsequently, starting from the first layer, we observe the pre-activation values (sum of the weighted spiking inputs) across $T_c$ timesteps, and set the firing threshold for neurons in that layer to be the $\rho$-th percentile value of the distribution of pre-activations, as performed in~\cite{lu2020exploring}.
We then set the firing thresholds for all other layers in the same way, and thus estimate proportionally balanced firing threshold values as in~\cite{sengupta2019going,rathi2021diet}.
Finally, we scale our initial estimates with a factor $\lambda<1$ in order to promote a better spike flow early on during finetuning, and make these firing thresholds $\{V_{th}^l\}_{l=1}^{L-1}$ \textit{trainable} during the finetuning process (see Appendix~\ref{ref:appendix_implementation} for details on hyperparameters).
\subsection{Adversarial Finetuning of Sparse SNNs}
We perform post-conversion robust finetuning of our sparse SNNs, similar to hybrid ANN-to-SNN conversion methods.
This is mainly necessary to calibrate the restructured spatio-temporal SNN batch-norm layers, and to adversarially finetune the weights and firing thresholds for a desired $T$.
We denote the model predictions for an adversarial and a clean example as $f(\boldsymbol{\tilde{x}};\mathbf{m} \odot \boldsymbol{\theta})$ and $f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta})$.
These values are estimated by the sparse SNN via the integrated non-leaky membrane potential of output neurons over $T$ timesteps.
Accordingly, we use a regularized robust finetuning objective that facilitates stability of the output neuron membrane potentials under adversarial attacks~\cite{zhang2019theoretically}:
\begin{equation}
\begin{aligned}
\min_{\boldsymbol{\theta}} \mathbb{E} \left[ \mathcal{L}\left(f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta}), y\right) + \beta \cdot \underset{\boldsymbol{\tilde x} \in \mathcal{B}_{\epsilon}^{\scriptscriptstyle\infty}(\boldsymbol{x})}{\text{max}}
D_{\text{KL}}\left(f(\boldsymbol{\tilde{x}};\mathbf{m} \odot \boldsymbol{\theta}) \| f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta})\right)
\right],
\label{eq:robust_training_snn}
\end{aligned}
\end{equation}
such that $\beta$ controls the robustness-accuracy trade-off during finetuning.
Specifically, we adversarially finetune parameters $\boldsymbol{\theta}$ which consist of $\{\boldsymbol{W}^l,\varphi^l,\omega^l\}_{l=1}^L$ and $\{ V_{th}^l\}_{l=1}^{L-1}$.
The computational load of finetuning is mainly determined by the method used for inner maximization to obtain $\boldsymbol{\tilde{x}}$, which can be performed efficiently via single-step attacks as conventionally done in SNN studies~\cite{ding2022snn,ding2024robust}.
Our finetuning objective is optimized via spike-based BPTT, which necessitates surrogate gradients to backpropagate the error through the non-differentiable function in Eq.~\eqref{eq:lif_spike}.
We use piecewise linear surrogate gradients via:
$\frac{\partial \boldsymbol{o}^l(t)}{\partial \boldsymbol{v}^l(t)} = \frac{1}{\gamma_w^2} \cdot \text{max}\left\{0, \gamma_w -|\boldsymbol{v}^l(t^{-}) - V^l_{th}| \right\},$
with $\gamma_w=1$ during training.
For attack evaluations, we consider adaptive SNN adversaries that can vary both $\gamma_w$ and the shape of the surrogate function to reliably estimate adversarial perturbations.
\textbf{Maintaining Sparsity During Finetuning:}
We preserve the transferred sparse ANN connectivity during the finetuning phase and keep the network connectivity frozen.
This is performed by applying the global pruning mask $\boldsymbol{m}$, consisting of per-layer binary masks $\{\boldsymbol{M}^l\}_{l=1}^L$ that were obtained during the robust ANN pruning phase, at each forward pass operation and weight update:
\begin{equation}
\boldsymbol{W}^l \leftarrow \boldsymbol{W}^l - \eta\left(\Delta \boldsymbol{W}^l \odot \boldsymbol{M}^l\right).
\label{eq:apply_sparsity_mask}
\end{equation}
Our resulting SNN therefore maintains the same sparsity as the pretrained ANN, but with new parameters $\boldsymbol{\theta'}$ which are adversarially finetuned via spike-based BPTT.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}
\subsection{Experimental Setup}
\textbf{Datasets and Models:}
We experimented with CIFAR-10/100 and TinyImageNet datasets, using VGG-11, VGG-16 and WideResNet architectures with depth 28 and widening factor of 4, i.e., WRN-28-4.
All SNNs considered in this study were run for $T=8$ timesteps with IF neurons ($\tau=1$).
\textbf{Adversarial Pretraining of Dense ANNs:}
We trained baseline ANNs via the TRADES loss~\cite{zhang2019theoretically} for 100 epochs, using 10-step PGD with $\epsilon=2/255$ to craft adversarial examples in the inner maximization stage.
For dense ANNs pretrained on CIFAR-10, additional pseudo-labeled data samples during AT were also utilized, which was shown to improve robustness~\cite{carmon2019unlabeled}.
\textbf{Pruning Adversarially Robust ANNs:}
We optimize robust weight importance scores for 20 epochs based on the configurations from HYDRA (for layerwise \textit{uniformly} sparse) and HARP (for layerwise \textit{non-uniformly} sparse) models.
We also perform naive LWM based pruning as a baseline.
We consider all dense and convolutional layer weights subject to unstructured pruning.
After pruning the ANNs to a desired sparsity ratio $\kappa$, we finetune sparse ANNs for 30 epochs following~\cite{sehwag2020hydra,zhao2022holistic}.
\textbf{Adversarial Finetuning of Sparse SNNs:}
We perform adversarial SNN finetuning following Eq~\eqref{eq:robust_training_snn} with BPTT for 80 epochs, using random-step FGSM (RFGSM)~\cite{tramer2017ensemble} with $\epsilon=2/255$ in the inner maximization stage.
We set $\beta=2$ in all experiments, unless specified otherwise.
During conversion, we initialized per-layer firing thresholds with the $\rho=99.7\%$ percentile of the pre-activation values that we observed over $T_c=100$ calibration timesteps, and use $\lambda=0.3$ as the scaling factor.
\textbf{Baseline Comparisons to End-to-End Sparse AT:}
As there are no existing solutions to this problem, we compare our conversion-based approach with end-to-end adversarial SNN training with a static sparse connectivity, using the same objective in Eq.~\eqref{eq:robust_training_snn}.
This connectivity was determined either randomly by uniform sampling, or set as the same sparse connectivity from the pretrained sparse ANN.
These networks were adversarially trained from scratch for at most 350 epochs.
\subsection{Evaluating Adversarial Robustness of SNNs}
The success of gradient-based attacks on SNNs strictly depend on the surrogate gradient used for BPTT~\cite{xu2022securing}.
The naive common practice is to use the same surrogate gradient for the attack as the one used during training~\cite{ding2022snn,ding2024enhancing}, which might lead to ineffective attacks and give a false sense of security for SNNs.
Accordingly, recent works have assessed SNN robustness using an \textit{ensemble} attack benchmark, where the adversary does not only utilize a single gradient approximation path, but via an ensemble of various surrogate gradient options during BPTT until successful~\cite{ozdenizci2024adversarially,liu2024enhancing} (see Appendix~\ref{ref:appendix_ensemble} for details).
This significantly increases attack effectiveness against SNNs and yields a worst case evaluation scenario~\cite{biggio2013security,carlini2019evaluating} (see Table~\ref{tab:ensemble_parts} for an ensemble attack analysis).
Accordingly, we evaluate SNNs against \textit{ensemble} attacks with FGSM~\cite{goodfellow2014explaining}, PGD~\cite{madry2017towards}, Auto-PGD with difference of logits ratio (APGD-DLR), and with cross-entropy loss (APGD-CE) \cite{croce2020reliable}.
Main PGD attacks were run with 10 steps (independently for each surrogate gradient in the ensemble), with step size $\alpha=2.5\times\epsilon/\text{\# steps}$.
In TinyImageNet experiments, we also evaluated SNNs against rate gradient approximation (RGA) attacks combined with 10-step PGD, which are designed for SNNs~\cite{bu2023rate}.
We evaluated black-box robustness with Square Attack~\cite{andriushchenko2020square}, using various numbers of limited queries.
\subsection{Estimation of Energy Consumption in SNNs}\label{sec:calculation_energy_efficiency}
Energy-efficiency in feedforward SNNs is tightly coupled with the overall spiking activity during inference.
Besides the memory-efficiency offered by sparse connectivities, we assess the impact of sparsity on the model's spiking activity, to verify that targeting sparsity does not degrade the energy-efficiency of the robust SNN as opposed to its dense counterpart due to higher spike rates.
Following the notation from~\cite{suetake2023synaptic}, we denote the number of \textit{active} outgoing connections from the $i$-th neuron in layer $l$ by $\psi_{l,i}$, i.e., the number of non-zero elements in the matrix $(\boldsymbol{W}^{l+1})_{*, i}$.
For densely connected feedforward SNNs with $\kappa=0$ this would result in:
\begin{align}
\psi_{l,i} &= \psi_l = c_{l+1} & \text{(linear layer),} \\
\psi_{l,i} &\simeq \psi_l = \frac{w_{l+1} h_{l+1}}{w_l h_l}c_{l+1}k^2_{l+1} & \text{(conv layer),}
\end{align}
where $c_l$ is the number of filters, $w_l, h_l$ are the feature map dimensions, and $k_l$ is the kernel width.
We use $\frac{w_{l+1} h_{l+1}}{w_l h_l}$ for normalization if pooling is performed between layers.
%
For sparse SNNs, we specifically take into account the exact connectivity structure and calculate all outgoing connections for every neuron to obtain $\psi_{l,i}$.
This yields an accurate estimate since only spikes over non-zero synaptic weights result in AC operations.
Subsequently, we estimate the energy consumption via:
\begin{equation}
E_{\text{SNN}} \defeq\sum_{l=1}^L\;T\cdot E_{\text{AC}} \cdot
\underset{\boldsymbol{x} \in \mathcal{X}}{\mathbb{E}}
\left[ \sum_{i=1}^{d_l} \psi_{l,i} o_{l,i} \right],
\label{eq:energy_snn}
\end{equation}
where $E_{\text{AC}}$ denotes the energy consumption of a single accumulate operation, $d_l$ is the number of neurons in the $l$-th layer, $o_{l,i}\in\{0,1\}$ are the output spikes emitted by neuron $i$ in the $l$-th layer, and $\mathbb{E}_{\boldsymbol{x} \in \mathcal{X}}[.]$ indicates taking the empirical expectation over the available test dataset $\mathcal{X}$.
It is important to note that we will use Eq.~\eqref{eq:energy_snn} to estimate \textit{relative} energy-efficiency gains of achieving sparsity in the SNNs, in relation to their dense SNN counterparts, in the context of preserving adversarial robustness under sparsity constraints.
We do not intend to compare SNNs with ANNs, since this would require some heuristic assumption on the relationship between $E_{\text{AC}}$ and $E_{\text{MAC}}$~\cite{horowitz20141}.
\begin{table*}[t]
\centering
\caption{Detailed evaluations of 90\% sparse VGG-16 models with layerwise uniform sparsity on CIFAR-10. We used white-box attacks with $\epsilon=2/255$ / $4/255$ / $8/255$, and black-box SquareAttack with $\epsilon=8/255$ using 500 / 1000 / 5000 queries. For SNNs, white-box attacks are implemented using a surrogate gradient ensemble. SquareAttack was run identically for SNNs and ANNs.}
\label{tab:baseline}
\scalebox{0.85}{
\begin{tabular}{l c c c l c}
\toprule
& \multicolumn{3}{c}{Sparse \& Robust SNN} & & \multirow{3}{*}[-4pt]{\parbox[t]{2.4cm}{\centering Sparse \&\\Robust ANN}} \\
\cmidrule(lr){2-4}
& \multicolumn{2}{c}{End-to-End Adv. Training} &
\multirow{2}{*}[-2pt]{\parbox[t]{2.9cm}{\centering \textbf{Conversion +\\Sparse FT (Ours)}}} \\
\cmidrule(lr){2-3}
& {\parbox[t]{2.5cm}{\centering Random Conn.}} & {\parbox[t]{2.5cm}{\centering ANN Conn.}} & \\
\cmidrule(lr){1-4}\cmidrule(lr){5-6}
{\parbox[t]{2.3cm}{Clean Acc.}} & 87.4 & 88.2 & \textbf{89.5} & {\parbox[t]{2.1cm}{Clean Acc.}} & 92.3 \\
\cmidrule(lr){1-4}\cmidrule(lr){5-6}
FGSM$_\text{ens}$ & 65.6 / 50.8 / 27.8 & 64.9 / 49.8 / 26.5 & \textbf{71.6} / \textbf{61.0} / \textbf{41.1} & FGSM & 81.3 / 67.4 / 42.5 \\
PGD$_\text{ens}$ & 62.4 / 42.2 / 11.9 & 61.1 / 40.1 / 9.70 & \textbf{69.9} / \textbf{54.7} / \textbf{26.1} & PGD & 79.8 / 59.5 / 22.2 \\
APGD-DLR$_\text{ens}$ & 58.3 / 40.8 / 12.8 & 57.9 / 38.8 / 10.4 & \textbf{65.2} / \textbf{52.0} / \textbf{26.9} & APGD-DLR & 79.4 / 58.5 / 21.2 \\
APGD-CE$_\text{ens}$ & 55.7 / 36.5 / 8.80 & 54.7 / 34.2 / 7.10
& \textbf{62.6} / \textbf{47.9} / \textbf{21.1} & APGD-CE & 79.5 / 57.9 / 19.3 \\
\midrule
SquareAttack & 47.9 / 42.4 / 32.2 & 49.8 / 43.8 / 33.9 & \textbf{52.3} / \textbf{46.5} / \textbf{36.3} & SquareAttack & 61.1 / 51.6 / 32.7 \\
\bottomrule
\end{tabular}}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experimental Results}
\subsection{Robust ANN-to-SNN Conversion Outperforms End-to-End Sparse AT}
\label{sec:conversion_vs_end_to_end}
We compare our approach with end-to-end AT of sparse SNNs, to empirically demonstrate the necessity of a novel ANN-to-SNN conversion-based solution, to the problem of achieving sparsity in robust SNNs.
Figure~\ref{fig:accs_during_training} shows that for 90\% sparse SNNs (layerwise uniform sparsity), our method outperforms both end-to-end AT models throughout training (i.e., clean/robust acc. at the end of training, E2E w/random conn.: 87.4/11.9, E2E w/ANN conn.: 88.2/9.7, Ours: 89.5/26.1).
Our results highlight the importance of the robust ANN based weight initialization alongside the sparse connectivity structure.
Importantly, we achieve this result with a more computationally
%
\begin{wrapfigure}{r}{8.4cm}
\vspace{.1cm}
\includegraphics[width=0.55\textwidth]{conversion_vs_e2e.pdf}
\caption{Comparing our method with end-to-end (E2E) AT, via clean (solid) and robust (dashed) accuracies evaluated with PGD$_\text{ens}$ at $\epsilon=8/255$ (90\% sparse VGG-16 SNN on CIFAR-10). Sparsity in E2E training was initialized either randomly, or by using the ANN connectivity as in our method.}
\label{fig:accs_during_training}
\end{wrapfigure}
%
efficient method (i.e., total wall-clock time: E2E SNN AT: 40.6 hours vs Ours: 24.5 hours, see Appendix~\ref{ref:appendix_implementation} for details).
This is due to the majority of our robust optimization process being performed in the ANN domain, without the computational overhead of AT with BPTT.
We demonstrate in Table~\ref{tab:baseline} that our conversion method is superior in terms of robustness across different attacks.
We obtained sparse SNNs that are more than twice as robust against strong adversaries, than E2E adversarially trained models (e.g., robust acc. under APGD-CE$_\text{ens}$ with $\epsilon=8/255$, E2E w/random conn.: 8.8\%, E2E w/ANN conn.: 7.1\%, Ours: 21.1\%).
Note that we tried adapting existing end-to-end regularized AT based SNN robustness methods~\cite{ding2022snn,liu2024enhancing} to improve performance of E2E trained models with 90\% sparsity.
However, we observed that such regularization schemes made the highly sparse training process sensitive to hyper-parameters and unstable, and we were not successful in adapting these methods to this problem.
\textbf{Robustness Against Black-box Attacks:}
Evaluations with white-box attacks on SNNs and ANNs are not necessarily identical or directly comparable, since SNN adversaries require a surrogate gradient ensemble approach.
However, Square Attack presents a fairer comparison since these query-based attacks are run identically without the use of gradients.
We observe that upon conversion our sparse SNN shows higher resilience after a high volume of available queries (robust acc. at 5000 queries, Our SNN: 36.3\%, ANN: 32.7\%).
For completeness, we also compared the 90\% sparse ANN and converted SNN with layerwise \textit{non-uniform} sparsity.
Our SNNs yielded stronger robustness than its ANN counterpart also in this setting (at 5000 queries, our SNN: 42.2\%, ANN: 37.0\%).
\subsection{Adversarial Robustness of Sparse SNNs}
\label{sec:results_robustness_sparsity}
Figure~\ref{fig:acc_cifar} presents detailed evaluations of our method with both layerwise uniform and non-uniform robust ANN pruning strategies, at various compression rates.
We also indicate a dense ANN-to-SNN conversion baseline, and LWM as a naive ANN pruning baseline.
Overall, we empirically validate that our approach generalizes well across different robust ANN pruning strategies, and we can reach any desired compression rate with an anticipated trade-off in clean and robust accuracies.
Notably, in our CIFAR-10 experiments, SNN robustness was not impacted up to 70\% sparsity (i.e., clean/robust acc., Dense SNN: 92.0/27.9, Uniform (70\%): 90.9/28.2).
We observed that layerwise non-uniform sparsity (shown with red lines) helps in maintaining robustness at high compression rates.
Specifically, we achieved clean/robust acc. under PGD$_\text{ens}$ for CIFAR-10 with 99\% sparsity (100$\times$~compression), LWM: 44.9/9.2, Uniform: 69.1/15.3, Non-Uniform: 85.2/22.9, and for CIFAR-100 with 90\% sparsity (10$\times$~compression), LWM: 56.6/17.6, Uniform: 63.8/20.7, Non-Uniform: 66.1/21.1 (see Table~\ref{tab:pgd_iterations} for PGD$_\text{ens}$ attack evaluations under various number of iterations).
In Appendix~\ref{appendix_annpt}, we investigate the influence of ANN pretraining on the robustness of the converted SNN.
We observed that more robust ANN weights result in better SNNs.
Specifically, we obtained 90\% sparse SNNs with clean/robust acc. of 83.2/40.0, when we performed heavier adversarial pretraining of the ANN using $\epsilon=8/255$, as opposed to the 89.5/26.1 from Table~\ref{tab:baseline}.
\begin{figure}[t]
\subfloat[CIFAR-10 with VGG-16]{
\includegraphics[width=0.48\textwidth]{acc_vgg16_cifar10.pdf}}\hspace{.25cm}
\subfloat[CIFAR-100 with WRN-28-4]{
\includegraphics[width=0.48\textwidth]{acc_wrn284_cifar100.pdf}}
\caption{Clean (solid lines) and robust (dashed lines) accuracies of converted SNNs evaluated with PGD$_\text{ens}$ at $\epsilon=8/255$ for CIFAR-10 and $\epsilon=4/255$ or CIFAR-100, for different sparsity levels. ANNs used for conversion were either dense, pruned with LWM, or layerwise uniformly or non-uniformly sparse through learned importance scores (see Appendix~\ref{appendix_evals} for numerical details of all results).}
\label{fig:acc_cifar}
\end{figure}
\subsection{Experiments on TinyImageNet}
We validate our method on the larger scale TinyImageNet dataset, which is generally not explored in the field of robust SNNs since existing methods that are based on end-to-end regularized AT with BPTT are not computationally feasible at this scale~\cite{ding2022snn,ding2024enhancing,ding2024robust,liu2024enhancing}.
In Table~\ref{tab:results_tinyimagenet} we present our results obtained by converting robustly pretrained ANNs with layerwise uniform sparsity.
Our sparse SNNs yield clean/robust accuracies that proportionally scale with the performance of the baseline ANN,
%
\begin{wraptable}{r}{8.95cm}
\caption{Evaluations on TinyImageNet with VGG-11 models at different layerwise uniform sparsity levels. We present robust accuracies (\%) under white-box attacks with perturbation budgets of $\epsilon=2/255$ and $\epsilon=4/255$.}\label{tab:results_tinyimagenet}
\scalebox{0.85}{
\begin{tabular}{l c cc cccc}
\toprule
& & \multicolumn{2}{c}{Robust ANN} & \multicolumn{3}{c}{Robust SNN \textbf{(Ours)}} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-7}
& & Clean & PGD & Clean & RGA & PGD$_\text{ens}$ \\
\midrule
\multicolumn{2}{c}{Dense} & 57.7 & 39.6 / 23.5 & 58.0 & 34.7 / 16.4 & 25.8 / 8.9 \\
\midrule
\parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{Sparse}}} &
50\% & 56.1 & 37.6 / 21.9 & 57.3 & 34.7 / 17.2 & 26.1 / 10.1 \\
& 70\% & 55.8 & 37.6 / 21.9 & 56.9 & 34.0 / 17.3 & 25.9 / 10.1 \\
& 90\% & 54.4 & 35.8 / 20.1 & 53.7 & 32.0 / 15.7 & 23.0 / 9.2 \\
& 95\% & 52.5 & 33.7 / 19.4 & 49.4 & 28.2 / 13.3 & 19.6 / 7.2 \\
\bottomrule
\end{tabular}}
\end{wraptable}
%
highlighting the importance of the pretrained ANN based sparse weight initialization, e.g., clean/robust acc. under PGD$_\text{ens}$ with $\epsilon=2/255$, our 50\% sparse SNN: 57.3/26.1 and 90\% sparse SNN: 53.7/23.0, while the clean/robust acc. under PGD with 50\% sparse ANN: 56.1/37.6 and 90\% sparse ANN: 54.4/35.8.
Notably, we also show that PGD$_\text{ens}$ attacks demonstrated a more rigorous evaluation baseline than RGA attacks~\cite{bu2023rate}, e.g., our 90\% sparse SNN with $\epsilon=4/255$, RGA: 15.7\%, PGD$_\text{ens}$: 9.2\%, and even FGSM$_\text{ens}$: 14.7\%.
\subsection{Impact of SNN Sparsity on Energy-Efficiency}
Table~\ref{tab:energy_comparisons} presents our spiking activity and energy efficiency analyses of VGG-16 models on CIFAR-10.
Firstly, we calculate the total number of spikes elicited by each SNN on average for a test sample.
Although the spiking activity of models appeared to increase with higher sparsity (see \#Spikes column), we also observed that our adversarially robust SNNs generally depicted high \textit{coding efficiency},
%
\begin{wraptable}{r}{8.6cm}
\caption{Relative energy efficiency estimates of our SNNs with respect to a dense SNN. \#Spikes column indicates the total \# of spikes elicited on average for a test sample.}\label{tab:energy_comparisons}
\scalebox{0.85}{
\begin{tabular}{c c c c c}
\toprule
& Sparsity & \#Spikes & $\frac{\#\text{Spikes}_{\text{DenseSNN}}}{\#\text{Spikes}_{\text{SparseSNN}}}$ & $\frac{E_{\text{DenseSNN}} }{E_{\text{SparseSNN}} }$ \\
\midrule
\parbox[t]{2cm}{{\parbox[t]{2.6cm}{\centering Dense SNN}}} & 0\% & 2.76$\times 10^{5}$ & 1.00 & \textbf{1.00} \\
\midrule
\parbox[t]{2cm}{\multirow{5}{*}[0pt]{\parbox[t]{2.6cm}{\centering Sparse SNN\\(Uniform)}}} & 50\% & 3.04$\times 10^{5}$ & 0.91 & \textbf{1.50} \\
& 60\% & 3.05$\times 10^{5}$ & 0.90 & \textbf{1.84} \\
& 70\% & 3.08$\times 10^{5}$ & 0.89 & \textbf{2.42} \\
& 80\% & 2.87$\times 10^{5}$ & 0.96 & \textbf{4.13} \\
& 90\% & 3.17$\times 10^{5}$ & 0.87 & \textbf{6.12} \\
\midrule
\parbox[t]{2cm}{\multirow{3}{*}[0pt]{\parbox[t]{2.6cm}{\centering Sparse SNN\\(Non-Uniform)}}} & 90\% & 3.02$\times 10^{5}$ & 0.91 & \textbf{2.15} \\
& 95\% & 3.07$\times 10^{5}$ & 0.90 & \textbf{3.10} \\
& 99\% & 3.14$\times 10^{5}$ & 0.88 & \textbf{8.61} \\
\bottomrule
\end{tabular}}
\end{wraptable}
%
individually.
Specifically, VGG-16 consists of 276,992 IF neurons operating for $T=8$ timesteps, which could induce a total of $\sim$2.2M possible spikes for a single inference.
However, the dense SNN (top row) was only approx. 12.5\% actively spiking ($2.76\times10^5/2.2\times10^6$), or similarly the 99\% sparse SNN (bottom row) was only 14.3\% actively spiking.
Considering the sparsity structure of each SNN and based on Eq.~\eqref{eq:energy_snn}, we then estimate the relative energy efficiency with respect to the dense SNN (see rightmost column in Table~\ref{tab:energy_comparisons}).
We observed that despite the increase in \#Spikes, sparsity in fact consistently yielded more energy-efficient models.
This was due to the decrease in the number of synaptic connections (hence less propagated spikes) largely outweighing the increase in spiking activity (see Appendix~\ref{sec:appendix_spikerates} for per-layer spike rates).
%
Overall, we obtained adversarially robust SNNs with layerwise uniform compression rates of up to 10$\times$, which were also up to $\sim$6.1$\times$ more energy efficient than a dense and robust SNN.
Our SNNs with non-uniform per-layer sparsities achieved up to 100$\times$ reduction in the number of parameters, with an estimated 8.6$\times$ energy efficiency increase relative to the dense SNN.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}
We introduce a solution to the problem of achieving sparsity in adversarially robust SNNs, which has not been investigated before.
We show that the sparse and robust network connectivity and weights optimized in the ANN domain can be effectively used in a novel hybrid ANN-to-SNN conversion algorithm with post-conversion sparse finetuning.
Our approach is superior to end-to-end AT of sparse SNNs in terms of performance and training efficiency, and achieves state-of-the-art robustness against rigorous adversaries.
Our energy consumption estimations reveal that the resulting SNNs are also more energy-efficient at high compression rates, than their dense counterparts.
Our robust SNN finetuning method can also be extended with dynamic robust and sparse training methods that enable rewiring of the transferred ANN connectivity~\cite{ozdenizci2021training}.
In this work, we focus on unstructured sparsity, since existing robust ANN pruning algorithms exclusively consider these methods.
One can also consider structured sparsity in the pretrained ANN, which might be more suitable for hardware oriented memory efficiency gains~\cite{he2017channel}, albeit with limited robustness capabilities~\cite{liebenwein2021lost}.
Our work fundamentally differs from recent SNN pruning methods, as these approaches do not consider robustness objectives~\cite{shi2024towards,yin2024workload}, and we perform robust pruning in the ANN domain.
We evaluate feedforward SNNs on standard machine learning tasks similar to previous SNN robustness studies~\cite{ding2022snn,ding2024robust,ding2024enhancing}, as opposed to spike-based data streams~\cite{marchisio2021dvs,liang2021exploring,buechel2022}.
This enables us to use traditional ANN architectures pretrained on the same task, which is needed for conversion.
Through this ANN, we can incorporate existing robust training and pruning schemes from the domain of ANNs to improve the robustness and structure the sparsity of the resulting SNNs.
Thus, our contribution is suitable to incorporate existing and future advances in the field of adversarial machine learning with traditional ANNs, towards energy and memory efficient, reliable AI applications.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Acknowledgments}
This research was funded in whole or in part by the Austrian Science Fund (FWF) [10.55776/COE12].
This work was supported by the Graz Center for Machine Learning (GraML), the NSF EFRI grant \#2318152, and the ``University SAL Labs'' initiative of Silicon Austria Labs (SAL) and its Austrian partner universities for applied fundamental research for electronic based systems.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{thebibliography}{63}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Szegedy et~al.(2013)Szegedy, Zaremba, Sutskever, Bruna, Erhan, Goodfellow, and Fergus]{szegedy2013intriguing}
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.
\newblock Intriguing properties of neural networks.
\newblock \emph{arXiv preprint arXiv:1312.6199}, 2013.
\bibitem[Madry et~al.(2017)Madry, Makelov, Schmidt, Tsipras, and Vladu]{madry2017towards}
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
\newblock Towards deep learning models resistant to adversarial attacks.
\newblock \emph{arXiv preprint arXiv:1706.06083}, 2017.
\bibitem[Nakkiran(2019)]{nakkiran2019adversarial}
Preetum Nakkiran.
\newblock Adversarial robustness may be at odds with simplicity.
\newblock \emph{arXiv preprint arXiv:1901.00532}, 2019.
\bibitem[Ye et~al.(2019)Ye, Xu, Liu, Cheng, Lambrechts, Zhang, Zhou, Ma, Wang, and Lin]{ye2019adversarial}
Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin.
\newblock Adversarial robustness vs. model compression, or both?
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, 2019.
\bibitem[Sehwag et~al.(2020)Sehwag, Wang, Mittal, and Jana]{sehwag2020hydra}
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana.
\newblock Hydra: Pruning adversarially robust neural networks.
\newblock \emph{Advances in Neural Information Processing Systems}, 33, 2020.
\bibitem[Zhao and Wressnegger(2022)]{zhao2022holistic}
Qi~Zhao and Christian Wressnegger.
\newblock Holistic adversarially robust pruning.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2022.
\bibitem[Kundu et~al.(2021)]{kundu2021hire}
Souvik Kundu et~al.
\newblock Hire-snn: Harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 5209--5218, 2021.
\bibitem[Ding et~al.(2022)Ding, Bu, Yu, Huang, and Liu]{ding2022snn}
Jianhao Ding, Tong Bu, Zhaofei Yu, Tiejun Huang, and Jian Liu.
\newblock {SNN-RAT}: Robustness enhanced spiking neural network through regularized adversarial training.
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 24780--24793, 2022.
\bibitem[Ding et~al.(2024{\natexlab{a}})Ding, Yu, Huang, and Liu]{ding2024enhancing}
Jianhao Ding, Zhaofei Yu, Tiejun Huang, and Jian~K Liu.
\newblock Enhancing the robustness of spiking neural networks with stochastic gating mechanisms.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~38, pages 492--502, 2024{\natexlab{a}}.
\bibitem[Liu et~al.(2024)Liu, Bu, Ding, Hao, Huang, and Yu]{liu2024enhancing}
Yujia Liu, Tong Bu, Jianhao Ding, Zecheng Hao, Tiejun Huang, and Zhaofei Yu.
\newblock Enhancing adversarial robustness in {SNN}s with sparse gradients.
\newblock In \emph{International Conference on Machine Learning}. PMLR, 2024.
\bibitem[{\"O}zdenizci and Legenstein(2024)]{ozdenizci2024adversarially}
Ozan {\"O}zdenizci and Robert Legenstein.
\newblock Adversarially robust spiking neural networks through conversion.
\newblock \emph{Transactions on Machine Learning Research}, 2024.
\bibitem[Roy et~al.(2019)]{roy2019towards}
Kaushik Roy et~al.
\newblock Towards spike-based machine intelligence with neuromorphic computing.
\newblock \emph{Nature}, 575\penalty0 (7784):\penalty0 607--617, 2019.
\bibitem[Davies et~al.(2021)Davies, Wild, Orchard, Sandamirskaya, Guerra, Joshi, Plank, and Risbud]{davies2021advancing}
Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A~Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh~R Risbud.
\newblock Advancing neuromorphic computing with {L}oihi: A survey of results and outlook.
\newblock \emph{Proceedings of the IEEE}, 109\penalty0 (5):\penalty0 911--934, 2021.
\bibitem[Wu et~al.(2018)Wu, Deng, Li, Zhu, and Shi]{wu2018spatio}
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi.
\newblock Spatio-temporal backpropagation for training high-performance spiking neural networks.
\newblock \emph{Frontiers in Neuroscience}, 12:\penalty0 331, 2018.
\bibitem[Wu et~al.(2019)Wu, Deng, Li, Zhu, Xie, and Shi]{wu2019direct}
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi.
\newblock Direct training for spiking neural networks: Faster, larger, better.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~33, pages 1311--1318, 2019.
\bibitem[Maass(1997)]{maass1997networks}
Wolfgang Maass.
\newblock Networks of spiking neurons: the third generation of neural network models.
\newblock \emph{Neural Networks}, 10\penalty0 (9):\penalty0 1659--1671, 1997.
\bibitem[Lee et~al.(2020)Lee, Sarwar, Panda, Srinivasan, and Roy]{lee2020enabling}
Chankyu Lee, Syed~Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy.
\newblock Enabling spike-based backpropagation for training deep neural network architectures.
\newblock \emph{Frontiers in Neuroscience}, page 119, 2020.
\bibitem[Neftci et~al.(2019)]{neftci2019surrogate}
Emre~O Neftci et~al.
\newblock Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks.
\newblock \emph{IEEE Signal Processing Magazine}, 36\penalty0 (6):\penalty0 51--63, 2019.
\bibitem[Diehl et~al.(2015)Diehl, Neil, Binas, Cook, Liu, and Pfeiffer]{diehl2015fast}
Peter~U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer.
\newblock Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing.
\newblock In \emph{International Joint Conference on Neural Networks (IJCNN)}, pages 1--8, 2015.
\bibitem[Rueckauer et~al.(2017)Rueckauer, Lungu, Hu, Pfeiffer, and Liu]{rueckauer2017conversion}
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu.
\newblock Conversion of continuous-valued deep networks to efficient event-driven networks for image classification.
\newblock \emph{Frontiers in Neuroscience}, 11:\penalty0 682, 2017.
\bibitem[Sengupta et~al.(2019)Sengupta, Ye, Wang, Liu, and Roy]{sengupta2019going}
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy.
\newblock Going deeper in spiking neural networks: Vgg and residual architectures.
\newblock \emph{Frontiers in Neuroscience}, 13:\penalty0 95, 2019.
\bibitem[Rathi et~al.(2020)Rathi, Srinivasan, Panda, and Roy]{rathi2020enabling}
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy.
\newblock Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation.
\newblock \emph{arXiv preprint arXiv:2005.01807}, 2020.
\bibitem[Rathi and Roy(2021)]{rathi2021diet}
Nitin Rathi and Kaushik Roy.
\newblock Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization.
\newblock \emph{IEEE Transactions on Neural Networks and Learning Systems}, 2021.
\bibitem[Horowitz(2014)]{horowitz20141}
Mark Horowitz.
\newblock Computing's energy problem (and what we can do about it).
\newblock In \emph{IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC)}, pages 10--14, 2014.
\bibitem[Goodfellow et~al.(2014)]{goodfellow2014explaining}
Ian~J Goodfellow et~al.
\newblock Explaining and harnessing adversarial examples.
\newblock \emph{arXiv preprint arXiv:1412.6572}, 2014.
\bibitem[Zhang et~al.(2019)Zhang, Yu, Jiao, Xing, El~Ghaoui, and Jordan]{zhang2019theoretically}
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El~Ghaoui, and Michael Jordan.
\newblock Theoretically principled trade-off between robustness and accuracy.
\newblock In \emph{International Conference on Machine Learning}, pages 7472--7482. PMLR, 2019.
\bibitem[Wang et~al.(2019)Wang, Zou, Yi, Bailey, Ma, and Gu]{wang2019improving}
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu.
\newblock Improving adversarial robustness requires revisiting misclassified examples.
\newblock In \emph{International Conference on Learning Representations}, 2019.
\bibitem[Tack et~al.(2022)Tack, Yu, Jeong, Kim, Hwang, and Shin]{tack2022consistency}
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung~Ju Hwang, and Jinwoo Shin.
\newblock Consistency regularization for adversarial robustness.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~36, pages 8414--8422, 2022.
\bibitem[Sharmin et~al.(2020)Sharmin, Rathi, Panda, and Roy]{sharmin2020inherent}
Saima Sharmin, Nitin Rathi, Priyadarshini Panda, and Kaushik Roy.
\newblock Inherent adversarial robustness of deep spiking neural networks: Effects of discrete input encoding and non-linear activations.
\newblock In \emph{European Conference on Computer Vision}, pages 399--414, 2020.
\bibitem[Liang et~al.(2022)Liang, Xu, Hu, Deng, and Xie]{liang2022toward}
Ling Liang, Kaidi Xu, Xing Hu, Lei Deng, and Yuan Xie.
\newblock Toward robust spiking neural network against adversarial perturbation.
\newblock \emph{Advances in Neural Information Processing Systems}, 35, 2022.
\bibitem[Sharmin et~al.(2019)Sharmin, Panda, Sarwar, Lee, Ponghiran, and Roy]{sharmin2019comprehensive}
Saima Sharmin, Priyadarshini Panda, Syed~Shakib Sarwar, Chankyu Lee, Wachirawit Ponghiran, and Kaushik Roy.
\newblock A comprehensive analysis on adversarial robustness of spiking neural networks.
\newblock In \emph{International Joint Conference on Neural Networks (IJCNN)}, pages 1--8. IEEE, 2019.
\bibitem[Marchisio et~al.(2020)Marchisio, Nanfa, Khalid, Hanif, Martina, and Shafique]{marchisio2020spiking}
Alberto Marchisio, Giorgio Nanfa, Faiq Khalid, Muhammad~Abdullah Hanif, Maurizio Martina, and Muhammad Shafique.
\newblock Is spiking secure? a comparative study on the security vulnerabilities of spiking and deep neural networks.
\newblock In \emph{International Joint Conference on Neural Networks (IJCNN)}, pages 1--8. IEEE, 2020.
\bibitem[El-Allami et~al.(2021)El-Allami, Marchisio, Shafique, and Alouani]{el2021securing}
Rida El-Allami, Alberto Marchisio, Muhammad Shafique, and Ihsen Alouani.
\newblock Securing deep spiking neural networks against adversarial attacks through inherent structural parameters.
\newblock In \emph{Design, Automation \& Test in Europe Conference \& Exhibition (DATE)}, pages 774--779. IEEE, 2021.
\bibitem[Xu et~al.(2022)Xu, Mahmood, Fang, Rathbun, Ding, and Wen]{xu2022securing}
Nuo Xu, Kaleel Mahmood, Haowen Fang, Ethan Rathbun, Caiwen Ding, and Wujie Wen.
\newblock Securing the spike: On the transferabilty and security of spiking neural networks to adversarial examples.
\newblock \emph{arXiv preprint arXiv:2209.03358}, 2022.
\bibitem[Ding et~al.(2024{\natexlab{b}})Ding, Pan, Liu, Yu, and Huang]{ding2024robust}
Jianhao Ding, Zhiyu Pan, Yujia Liu, Zhaofei Yu, and Tiejun Huang.
\newblock Robust stable spiking neural networks.
\newblock In \emph{International Conference on Machine Learning}. PMLR, 2024{\natexlab{b}}.
\bibitem[Han et~al.(2015)Han, Pool, Tran, and Dally]{han2015learning}
Song Han, Jeff Pool, John Tran, and William Dally.
\newblock Learning both weights and connections for efficient neural network.
\newblock \emph{Advances in Neural Information Processing Systems}, 28, 2015.
\bibitem[Ramanujan et~al.(2020)Ramanujan, Wortsman, Kembhavi, Farhadi, and Rastegari]{ramanujan2020}
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari.
\newblock What's hidden in a randomly weighted neural network?
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 11893--11902, 2020.
\bibitem[Zheng et~al.(2021)Zheng, Wu, Deng, Hu, and Li]{zheng2021going}
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li.
\newblock Going deeper with directly-trained larger spiking neural networks.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~35, pages 11062--11070, 2021.
\bibitem[Lu and Sengupta(2020)]{lu2020exploring}
Sen Lu and Abhronil Sengupta.
\newblock Exploring the connection between binary and spiking neural networks.
\newblock \emph{Frontiers in Neuroscience}, 14:\penalty0 535, 2020.
\bibitem[Carmon et~al.(2019)Carmon, Raghunathan, Schmidt, Duchi, and Liang]{carmon2019unlabeled}
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John~C Duchi, and Percy~S Liang.
\newblock Unlabeled data improves adversarial robustness.
\newblock \emph{Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem[Tram{\`e}r et~al.(2017)Tram{\`e}r, Kurakin, Papernot, Goodfellow, Boneh, and McDaniel]{tramer2017ensemble}
Florian Tram{\`e}r, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.
\newblock Ensemble adversarial training: Attacks and defenses.
\newblock \emph{arXiv preprint arXiv:1705.07204}, 2017.
\bibitem[Biggio et~al.(2013)]{biggio2013security}
Battista Biggio et~al.
\newblock Security evaluation of pattern classifiers under attack.
\newblock \emph{IEEE Transactions on Knowledge and Data Engineering}, 26\penalty0 (4):\penalty0 984--996, 2013.
\bibitem[Carlini et~al.(2019)Carlini, Athalye, Papernot, Brendel, Rauber, Tsipras, Goodfellow, Madry, and Kurakin]{carlini2019evaluating}
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin.
\newblock On evaluating adversarial robustness.
\newblock \emph{arXiv preprint arXiv:1902.06705}, 2019.
\bibitem[Croce and Hein(2020)]{croce2020reliable}
Francesco Croce and Matthias Hein.
\newblock Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.
\newblock In \emph{International Conference on Machine Learning}, pages 2206--2216. PMLR, 2020.
\bibitem[Bu et~al.(2023)Bu, Ding, Hao, and Yu]{bu2023rate}
Tong Bu, Jianhao Ding, Zecheng Hao, and Zhaofei Yu.
\newblock Rate gradient approximation attack threats deep spiking neural networks.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 7896--7906, 2023.
\bibitem[Andriushchenko et~al.(2020)Andriushchenko, Croce, Flammarion, and Hein]{andriushchenko2020square}
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein.
\newblock Square attack: a query-efficient black-box adversarial attack via random search.
\newblock In \emph{European Conference on Computer Vision (ECCV)}, pages 484--501. Springer, 2020.
\bibitem[Suetake et~al.(2023)Suetake, Ushimaru, Saiin, and Sawada]{suetake2023synaptic}
Kazuma Suetake, Takuya Ushimaru, Ryuji Saiin, and Yoshihide Sawada.
\newblock Synaptic interaction penalty: Appropriate penalty term for energy-efficient spiking neural networks.
\newblock \emph{Transactions on Machine Learning Research}, 2023.
\bibitem[{\"O}zdenizci and Legenstein(2021)]{ozdenizci2021training}
Ozan {\"O}zdenizci and Robert Legenstein.
\newblock Training adversarially robust sparse networks via {B}ayesian connectivity sampling.
\newblock In \emph{International Conference on Machine Learning}, pages 8314--8324. PMLR, 2021.
\bibitem[He et~al.(2017)]{he2017channel}
Yihui He et~al.
\newblock Channel pruning for accelerating very deep neural networks.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer Vision}, pages 1389--1397, 2017.
\bibitem[Liebenwein et~al.(2021)Liebenwein, Baykal, Carter, Gifford, and Rus]{liebenwein2021lost}
Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus.
\newblock Lost in pruning: The effects of pruning neural networks beyond test accuracy.
\newblock \emph{Proceedings of Machine Learning and Systems}, 3:\penalty0 93--138, 2021.
\bibitem[Shi et~al.(2024)Shi, Ding, Hao, and Yu]{shi2024towards}
Xinyu Shi, Jianhao Ding, Zecheng Hao, and Zhaofei Yu.
\newblock Towards energy efficient spiking neural networks: An unstructured pruning framework.
\newblock In \emph{International Conference on Learning Representations (ICLR)}, 2024.
\bibitem[Yin et~al.(2024)Yin, Kim, Li, Moitra, Satpute, Hambitzer, and Panda]{yin2024workload}
Ruokai Yin, Youngeun Kim, Yuhang Li, Abhishek Moitra, Nitin Satpute, Anna Hambitzer, and Priyadarshini Panda.
\newblock Workload-balanced pruning for sparse spiking neural networks.
\newblock \emph{IEEE Transactions on Emerging Topics in Computational Intelligence}, 2024.
\bibitem[Marchisio et~al.(2021)Marchisio, Pira, Martina, Masera, and Shafique]{marchisio2021dvs}
Alberto Marchisio, Giacomo Pira, Maurizio Martina, Guido Masera, and Muhammad Shafique.
\newblock Dvs-attacks: Adversarial attacks on dynamic vision sensors for spiking neural networks.
\newblock In \emph{International Joint Conference on Neural Networks (IJCNN)}, pages 1--9. IEEE, 2021.
\bibitem[Liang et~al.(2021)Liang, Hu, Deng, Wu, Li, Ding, Li, and Xie]{liang2021exploring}
Ling Liang, Xing Hu, Lei Deng, Yujie Wu, Guoqi Li, Yufei Ding, Peng Li, and Yuan Xie.
\newblock Exploring adversarial attack in spiking neural networks with spike-compatible gradient.
\newblock \emph{IEEE Transactions on Neural Networks and Learning Systems}, 2021.
\bibitem[Büchel et~al.(2022)Büchel, Lenz, Hu, Sheik, and Sorbaro]{buechel2022}
Julian Büchel, Gregor Lenz, Yalun Hu, Sadique Sheik, and Martino Sorbaro.
\newblock Adversarial attacks on spiking convolutional neural networks for event-based vision.
\newblock \emph{Frontiers in Neuroscience}, 16, 2022.
\bibitem[Krizhevsky(2009)]{Krizhevsky:2009}
Alex Krizhevsky.
\newblock Learning multiple layers of features from tiny images.
\newblock \emph{Technical Report, University of Toronto}, 2009.
\bibitem[Le and Yang(2015)]{le2015tiny}
Ya~Le and Xuan Yang.
\newblock Tiny {I}mage{N}et visual recognition challenge, 2015.
\bibitem[Simonyan and Zisserman(2015)]{Simonyan:2015}
Karen Simonyan and Andrew Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock In \emph{International Conference on Learning Representations}, 2015.
\bibitem[Zagoruyko(2016)]{Zagoruyko:2016}
Sergey Zagoruyko.
\newblock Wide residual networks.
\newblock \emph{arXiv preprint arXiv:1605.07146}, 2016.
\bibitem[Bellec et~al.(2018)Bellec, Salaj, Subramoney, Legenstein, and Maass]{bellec2018long}
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass.
\newblock Long short-term memory and learning-to-learn in networks of spiking neurons.
\newblock \emph{Advances in Neural Information Processing Systems}, 31, 2018.
\bibitem[Shrestha and Orchard(2018)]{shrestha2018slayer}
Sumit~B Shrestha and Garrick Orchard.
\newblock Slayer: Spike layer error reassignment in time.
\newblock \emph{Advances in Neural Information Processing Systems}, 31, 2018.
\bibitem[Bengio et~al.(2013)]{bengio2013estimating}
Yoshua Bengio et~al.
\newblock Estimating or propagating gradients through stochastic neurons for conditional computation.
\newblock \emph{arXiv preprint arXiv:1308.3432}, 2013.
\bibitem[Athalye et~al.(2018)Athalye, Carlini, and Wagner]{athalye2018obfuscated}
Anish Athalye, Nicholas Carlini, and David Wagner.
\newblock Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.
\newblock In \emph{International Conference on Machine Learning}, pages 274--283. PMLR, 2018.
\end{thebibliography}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix
\makeatletter
\setcounter{table}{0}
\setcounter{figure}{0}
\renewcommand\thetable{A\@arabic\c@table}
\renewcommand \thefigure{A\@arabic\c@figure}
\makeatother
\section{Details on the Experimental Setup}
\label{sec:exp_details}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Datasets and Models}
\label{ref:appendix_datasets}
We experimented with CIFAR-10/100 and TinyImageNet datasets.
CIFAR datasets both consist of 32$\times$32 dimensional 50,000 training and 10,000 test images, from 10 and 100 classes respectively~\cite{Krizhevsky:2009}.
TinyImageNet dataset consists of 64$\times$64 dimensional 100,000 training and 10,000 test images from 200 classes~\cite{le2015tiny}.
We use conventional data augmentation approaches involving random cropping of images into 32$\times$32 dimensions by padding zeros for at most 4 pixels around it (or random resized crop to 64$\times$64 for TinyImageNet), and randomly performing a horizontal flip.
We use VGG-11, VGG-16~\citep{Simonyan:2015} and WideResNet~\citep{Zagoruyko:2016} architecture with depth 28 and width 4 (i.e., WRN-28-4).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Implementation \& Training Configurations}
\label{ref:appendix_implementation}
Our proposed adversarially robust and sparse ANN-to-SNN conversion methodology is outlined in Algorithm~\ref{alg:robust_snn_conversion}.
It is an extension of the robust ANN-to-SNN conversion method introduced for densely connected neural network models~\cite{ozdenizci2024adversarially}, and the implementations are publicly available at: \url{https://github.com/IGITUGraz/RobustSNNConversion}.
\textbf{Adversarial Pretraining of Dense ANNs:}
We perform adversarial training via the TRADES~\cite{zhang2019theoretically} using 10-step PGD with $l_{\infty}$-norm bounded perturbations of magnitude $\epsilon=2/255$.
We set the TRADES loss trade-off parameter $\lambda_{\text{TRADES}}=2$ in the main results.
We explore stronger AT with the baseline ANN in later ablation experiments with higher $\epsilon$ and $\lambda_{\text{TRADES}}=6$ in Table~\ref{tab:ann_eps}.
We pretrain the ANNs for 100 epochs via momentum SGD, with learning rate of 0.1 and a weight decay of 0.0001.
\textbf{Pruning Adversarially Robust ANNs:}
For layerwise uniformly and non-uniformly robust pruning, we follow the training configurations from HYDRA~\cite{sehwag2020hydra} and HARP~\cite{zhao2022holistic}, respectively.
We perform robust importance score optimization using the same dense ANN training objective for 20 epochs.
Based on these learned importance scores, we prune the model for a given $\kappa$.
Afterwards, we adversarially finetune sparse ANNs for 30 epochs, using momentum SGD with a learning rate of 0.01 and a weight decay of 0.0001~\cite{sehwag2020hydra}.
\textbf{Adversarial Finetuning of Sparse SNNs:}
During conversion, we estimate per-layer firing thresholds with the use of 10 training mini-batches of size 64.
We generate these inputs via direct coding for $T_c=100$, and observe the pre-activation values~\cite{rathi2021diet}.
We choose our initial estimates as the maximum value in the $\rho=99.7\%$ percentile of the distribution of observed values, similarly to~\cite{lu2020exploring,rathi2021diet}.
We use $\lambda=0.3$ as the scaling factor on these initial estimates, while setting the trainable firing thresholds.
Adversarial end-to-end SNN training baselines were implemented with sparse models initialized with $V_{th}^l=1$ for all layers, similar to~\cite{ding2022snn,ding2024enhancing}.
During conversion, we re-use the pretrained ANN batch-norm parameters for the SNN, but now estimate the mean and variance moving average statistics from scratch during robust SNN finetuning phase on the spiking data.
We set the simulation timesteps $T=8$ and consider non-leaky IF neurons ($\tau=1$) in all SNNs.
We perform robust and sparse SNN finetuning for 80 epochs, using momentum SGD with a initial learning rate of 0.001, a weight decay of 0.0001, and cosine annealing based learning rate schedulers throughout.
We use random-step FGSM (RFGSM)~\cite{tramer2017ensemble} based adversarial examples during robust and sparse finetuning of all SNN models, using an $l_{\infty}$-bounded perturbation budget of $\epsilon=2/255$, on the regularizer part of our finetuning objective function $\mathcal{L}_{\text{RFGSM}}=D_{\text{KL}}\left(f(\boldsymbol{\tilde{x}};\mathbf{m} \odot \boldsymbol{\theta}) \| f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta})\right)$.
\textbf{Computational Overhead:}
Since our method also involves an adversarial finetuning stage, it partially suffers from the same computational burden of AT with BPTT.
To obtain our models, we perform 100 epochs of dense ANN adversarial training, 20 epochs of robust importance score optimization, and 30 epochs of sparse ANN finetuning.
After ANN-to-SNN conversion, we perform adversarial finetuning of the sparse SNN via BPTT for 80 epochs.
Given this experimental setting, wall-clock training time comparisons on a single NVIDIA Quadro RTX 8000 GPU are: baseline ANN adversarial training (100 epochs) + robust ANN pruning (50 epochs) + robust SNN finetuning (80 epochs): $\sim$24.5 hours, whereas end-to-end SNN adversarial training for 350 epochs: $\sim$40.6 hours).
It is important to note that SNN-based parameter updates with AT is significantly more time consuming due to the temporal dimension and the need to unroll the network for BPTT also to compute adversarial examples.
However, since the majority of our parameter optimization is performed in the ANN domain, our method eventually results in being the more efficient way to achieve sparsity and adversarial robustness with SNNs.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{algorithm}
\caption{Robust and Sparse ANN-to-SNN Conversion}
\label{alg:robust_snn_conversion}
\begin{algorithmic}[1]
\STATE \textbf{Input:} Dataset $\mathcal{D}$, adversarially pretrained sparse ANN parameters $\{\boldsymbol{W}^l, \varphi^l, \omega^l\}_{l=1}^L$, number of calibration samples, calibration sequence length $T_c$, percentile $\rho$, threshold scaling factor $\lambda$, number of finetuning iterations, simulation timesteps $T$, membrane leak factor $\tau$, trade-off parameter $\beta$, attack perturbation strength $\epsilon$.
\STATE {\bfseries Output:} Robust and sparse SNN $f$ with parameters $\boldsymbol{\theta'}$ consisting of $\{\boldsymbol{W}^l,\varphi^l,\omega^l\}_{l=1}^L$ and $\{V_{th}^l\}_{l=1}^{L-1}$.
\vspace{.05cm}
\STATEx $\triangleright$ Converting adversarially pretrained sparse ANN
\STATE \textbf{Initialize: } Set weights of SNN $f$ directly from adversarially pretrained ANN parameters $\{\boldsymbol{W}^l, \varphi^l, \omega^l\}_{l=1}^L$. Define spiking LIF neurons for the SNN layers with $\tau$.
\FOR{$l = 1$ \textbf{to} $L - 1$}
\FOR{$c = 1$ \textbf{to} \#calibration\_samples}
\FOR{t = 1 \textbf{to} $T_c$}
\STATE $\boldsymbol{a}^l \leftarrow$ Store pre-activation values at layer $l$ during forward pass with direct input coding \
\ENDFOR
\IF{max$[\rho\text{-percentile of the dist. in } \boldsymbol{a}^l] > V_{th}^l$}
\STATE $V_{th}^l = \text{max}[\rho\text{-percentile of the dist. in } \boldsymbol{a}^l]$
\ENDIF
\ENDFOR
\ENDFOR
\vspace{.05cm}
\STATEx $\triangleright$ Initialize trainable firing threshold
\FOR{$l = 1$ \textbf{to} $L - 1$}
\STATE $V_{th}^l \leftarrow \lambda \cdot V_{th}^l$
\ENDFOR
\vspace{.05cm}
\STATEx $\triangleright$ Initialize binary sparsity mask $\textbf{M}$
\FOR{$l = 1$ \textbf{to} $L$}
\STATE $\textbf{M}^l_{j,i} \leftarrow {\begin{cases}1 \quad \text{if} \quad &\textbf{W}^l_{j,i}\neq 0\\0 \quad \text{if} \quad & \textbf{W}^l_{j,i} = 0\end{cases}}$
\ENDFOR
\vspace{.05cm}
\STATEx $\triangleright$ Robust and sparse finetuning of SNN after conversion
\FOR{$i = 1$ \textbf{to} \#finetuning\_iterations}
\STATE Sample a mini-batch of $(\boldsymbol{x}, y) \sim \mathcal{D}$
\STATE $\boldsymbol{\tilde x}\leftarrow$ Compute via RFGSM-based inner max. with $\epsilon$
\STATE $f(\boldsymbol{\tilde{x}};\mathbf{m} \odot \boldsymbol{\theta})\leftarrow$ Compute via direct coded $\boldsymbol{\tilde{x}}$ for $T$
\STATE $f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta})\leftarrow$ Compute via direct coded $\boldsymbol{x}$ for $T$
\STATE $\mathcal{L}_{\text{reg}}(\boldsymbol{x},\boldsymbol{\tilde x})\leftarrow D_{\text{KL}}\left(f(\boldsymbol{\tilde{x}};\mathbf{m} \odot \boldsymbol{\theta}) \| f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta})\right)$
\STATE $\mathcal{L}_{\text{robust}}\leftarrow\mathcal{L}(f(\boldsymbol{x};\mathbf{m} \odot \boldsymbol{\theta})) + \beta \cdot \mathcal{L}_{\text{reg}}(\boldsymbol{x},\boldsymbol{\tilde x})$
\STATE $\Delta \boldsymbol{W}^l \leftarrow \sum_t \frac{\partial \mathcal{L}_{\text{robust}}}{\partial \boldsymbol{W}^l}$
\STATE $\Delta \varphi^l \leftarrow \sum_t \frac{\partial \mathcal{L}_{\text{robust}}}{\partial \varphi^l}$
\STATE $\Delta \omega^l \leftarrow \sum_t \frac{\partial \mathcal{L}_{\text{robust}}}{\partial \omega^l}$
\STATE $\Delta V_{th}^l \leftarrow \sum_t \frac{\partial \mathcal{L}_{\text{robust}}}{\partial V_{th}^l}$
\vspace{.05cm}
\STATEx $\triangleright$ Maintain sparse weight connectivity via masks
\FOR{$l = 1$ \textbf{to} $L$}
\STATE $\boldsymbol{W}^l \leftarrow \boldsymbol{W}^l - \eta\left(\Delta \boldsymbol{W}^l \odot \boldsymbol{M}^l\right)$
\ENDFOR
\ENDFOR
\end{algorithmic}
\end{algorithm}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Ensemble Adversarial Attacks}
\label{ref:appendix_ensemble}
We consider adaptive adversaries that implement an ensemble SNN attack strategy as introduced in~\cite{ozdenizci2024adversarially}, where the adversary does not only utilize a single input gradient approximation path but via an ensemble of various surrogate gradient options during BPTT.
The ensemble consists of the piecewise linear function~\cite{bellec2018long} with $\gamma_w\in\{0.25,0.5,1.0,2.0,3.0\}$:
\begin{equation}
\frac{\partial\boldsymbol{o}^l(t)}{\partial \boldsymbol{v}^l(t)}=\frac{1}{\gamma_w^{2}}\cdot\max\{0,\gamma_w-|\boldsymbol{v}^l(t^{-}) - V^l_{th}|\},
\label{eq:pcw}
\end{equation}
the exponential surrogate gradient function~\cite{shrestha2018slayer} with $(\gamma_d,\gamma_s)\in\{(0.3,0.5),(0.3,1.0),(0.3,2.0),$
$(1.0,0.5),(1.0,1.0),(1.0,2.0)\}$:
\begin{equation}
\frac{\partial\boldsymbol{o}^l(t)}{\partial \boldsymbol{v}^l(t)}=\gamma_d\cdot\exp{\left(-\gamma_s\cdot|\boldsymbol{v}^l(t^{-}) - V^l_{th}|\right)},
\label{eq:exp}
\end{equation}
and the rectangular surrogate gradient function~\cite{wu2018spatio} with $\gamma_w\in\{0.25,0.5,1.0,2.0,4.0\}$:
\begin{equation}\frac{\partial\boldsymbol{o}^l(t)}{\partial \boldsymbol{v}^l(t)}=\frac{1}{\gamma_w}\cdot \text{sign}\left(|\boldsymbol{v}^l(t^{-}) - V^l_{th}|<\frac{\gamma_w}{2}\right).
\label{eq:rect}
\end{equation}
We also included the straight-through estimator (STE), where an identity function is used as the surrogate gradient for all spiking neurons during backpropagation~\cite{bengio2013estimating}, backward pass through rate (BPTR), which performs a differentiable approximation by taking the derivative of the spike functions directly from the average firing rate of the neurons between layers~\cite{ding2022snn}, and a conversion-based approximation, where spiking neurons are replaced with ReLUs during BPTT~\cite{sharmin2019comprehensive}.
We present a detailed decomposition of the individual components of the ensemble in Table~\ref{tab:ensemble_parts}.
Notably, we observed that altering the shape and parameters of the surrogate gradient can influence the success of the overall adversary on different test samples.
Several SNN robustness studies simply
consider the only naive adversary case, which only utilizes Eq.~\eqref{eq:pcw} with $\gamma_w=1$, i.e., identical surrogate gradient function to the one used during SNN training.
\begin{table*}[t]
\centering
\caption{Detailed robust accuracy results for the individual components of an FGSM ensemble attack with $\epsilon=8/255$ on VGG-16 models with 90\% sparsity. Strongest individual attack components in the ensemble against each model are underlined.}
\label{tab:ensemble_parts}
\scalebox{0.85}{
\begin{tabular}{l l c c c}
\toprule
& & \multicolumn{2}{c}{End-to-End Adv. Training} &
\multirow{2}{*}[-2pt]{\parbox[t]{2.9cm}{\centering \textbf{Conversion +\\Sparse FT (Ours)}}} \\
\cmidrule(lr){3-4}
& & {\parbox[t]{2.5cm}{\centering Random Conn.}} & {\parbox[t]{2.5cm}{\centering ANN Conn.}} & \\
\midrule
& Clean Acc. & 87.4 & 88.2 & \textbf{89.5} \\
& FGSM$_\text{ens}$ & 27.8 & 26.5 & \textbf{41.1} \\
\midrule
\parbox[t]{2mm}{\multirow{5}{*}{\rotatebox[origin=c]{90}{Pcw. Linear}}} & $\gamma_w = 1.0$ & 38.79 & 38.30 & \textbf{54.81} \\
& $\gamma_w = 2.0$ & 54.49 & 54.44 & \textbf{68.01} \\
& $\gamma_w = 3.0$ & 67.60 & 67.13 & \textbf{75.65} \\
& $\gamma_w = 0.5$ & 44.18 & 42.51 & \textbf{54.87} \\
& $\gamma_w = 0.25$ & 81.28 & 82.12 & \textbf{84.04} \\
\midrule
\parbox[t]{2mm}{\multirow{6}{*}{\rotatebox[origin=c]{90}{Exponential}}} & $(\gamma_d,\gamma_s) = (0.3,0.5)$ & 68.10 & 68.33 & \textbf{76.40} \\
& $(\gamma_d,\gamma_s) = (0.3,1.0)$ & 48.15 & 47.26 & \textbf{62.90} \\
& $(\gamma_d,\gamma_s) = (0.3,2.0)$ & \underline{34.29} & \underline{33.61} & \underline{\textbf{50.40}} \\
& $(\gamma_d,\gamma_s) = (1.0,0.5)$ & 78.15 & 79.12 & \textbf{82.38} \\
& $(\gamma_d,\gamma_s) = (1.0,1.0)$ & 56.36 & 56.35 & \textbf{69.66} \\
& $(\gamma_d,\gamma_s) = (1.0,2.0)$ & 37.01 & 36.38 & \textbf{53.44} \\
\midrule
\parbox[t]{2mm}{\multirow{5}{*}{\rotatebox[origin=c]{90}{Rectangular}}} & $\gamma_w = 0.25$ & 85.47 & 85.96 & \textbf{87.82} \\
& $\gamma_w = 0.5$ & 69.23 & 66.97 & \textbf{70.84} \\
& $\gamma_w = 1.0$ & 46.51 & 44.56 & \textbf{57.32} \\
& $\gamma_w = 2.0$ & 59.25 & 59.52 & \textbf{70.93} \\
& $\gamma_w = 4.0$ & 78.66 & 79.60 & \textbf{83.28} \\
\midrule
\multicolumn{2}{l}{Straight-Through Estimation} & 85.78 & 86.58 & \textbf{88.83} \\
\multicolumn{2}{l}{Backward Pass Through Rate} & 42.80 & 41.15 & \textbf{57.20} \\
\multicolumn{2}{l}{Conversion-based Approx.} & 80.73 & 80.61 & \textbf{70.84} \\
\toprule
\end{tabular}}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Further Experimental Results}
\label{sec:further_experiments}
\makeatletter
\setcounter{table}{0}
\setcounter{figure}{0}
\renewcommand\thetable{B\@arabic\c@table}
\renewcommand \thefigure{B\@arabic\c@figure}
\makeatother
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluations of Sparse ANNs and SNNs}
\label{appendix_evals}
We present the numerical evaluation details corresponding to the Figure 2 of the main manuscript, in Tables~\ref{tab:detail_cifar10} and~\ref{tab:detail_cifar100}.
These tables also include evaluations of the robust and sparse ANN models used for conversion in obtaining the corresponding robust and sparse SNNs.
Note that ANNs are evaluated with standard PGD attacks (10-steps) only for completeness.
Our primary goal was to perform comparative analysis of sparse SNN models under ensemble attacks.
Our first observation is that robust ANN pruning with learned importance scores significantly outperforms LWM-based pruning in many aspects of the resulting SNN, particularly in terms of benign performance at high model compression rates.
Mainly, we successfully achieve models with very high compression by using layerwise non-uniform sparsity structures, e.g., in Table~\ref{tab:detail_cifar10} clean/robust acc. under PGD$_\text{ens}$ with $\epsilon=8/255$ at 99\% SNN sparsity, LWM: 44.9/9.2, Uniform: 69.1/15.3, Non-Uniform: 85.2/22.9.
\begin{table*}[p]
\centering
\caption{Detailed evaluations of robust and sparse VGG-16 models on CIFAR-10, for different robust ANN pruning approaches and sparsity levels. Robust SNNs are obtained with our conversion method using the corresponding baseline ANNs.}
\label{tab:detail_cifar10}
\scalebox{0.85}{
\begin{tabular}{c c c c c c c c c c c}
\toprule
& \multirow{2}{*}[-2pt]{Sparsity} & \multicolumn{2}{c}{Robust ANN} & \multicolumn{3}{c}{\textbf{Robust SNN (Ours)}} \\
\cmidrule(lr){3-4}\cmidrule(lr){5-7}
& & Clean & PGD & Clean & FGSM$_\text{ens}$ & PGD$_\text{ens}$ \\
\midrule
\parbox[t]{2.5cm}{{\parbox[t]{2.5cm}{\centering Dense NN}}} & 0\% & 93.1 & 82.2 / 63.9 / 25.2 & 92.0 & 76.3 / 66.2 / 45.8 & 74.9 / 59.8 / 27.9 \\
\midrule
\parbox[t]{2.5cm}{\multirow{7}{*}[0pt]{\parbox[t]{2.5cm}{\centering Sparse NN\\(LWM)}}} & 10\% & 93.2 & 82.2 / 64.3 / 25.5 & 91.8 & 75.7 / 65.9 / 45.5 & 74.1 / 59.4 / 27.7 \\
& 30\% & 93.2 & 82.3 / 63.8 / 25.7 & 92.0 & 76.2 / 65.9 / 45.7 & 74.4 / 59.3 / 28.8 \\
& 50\% & 93.2 & 82.1 / 63.6 / 25.3 & 91.8 & 76.3 / 66.0 / 45.9 & 74.8 / 60.0 / 28.4 \\
& 70\% & 92.8 & 81.5 / 62.3 / 24.5 & 92.0 & 77.3 / 67.3 / 47.3 & 75.3 / 60.9 / 30.0 \\
& 90\% & 91.2 & 77.8 / 56.6 / 20.1 & 88.7 & 69.9 / 59.5 / 40.3 & 68.0 / 54.2 / 26.1 \\
& 95\% & 88.7 & 73.4 / 51.1 / 16.1 & 78.7 & 54.2 / 46.1 / 31.9 & 52.6 / 42.0 / 23.0 \\
& 99\% & 72.1 & 50.7 / 31.4 / 8.20 & 44.9 & 18.8 / 15.8 / 11.2 & 17.9 / 15.0 / 9.20 \\
\midrule
\parbox[t]{2.5cm}{\multirow{7}{*}[0pt]{\parbox[t]{2.5cm}{\centering Sparse NN\\(Uniform)}}} & 10\% & 93.2 & 82.3 / 64.2 / 25.6 & 91.2 & 75.3 / 64.8 / 43.8 & 73.6 / 59.0 / 28.1 \\
& 30\% & 93.3 & 82.3 / 63.9 / 25.5 & 90.8 & 74.6 / 64.4 / 44.8 & 72.5 / 57.9 / 27.9 \\
& 50\% & 93.2 & 81.9 / 63.0 / 25.3 & 90.7 & 74.7 / 64.8 / 44.6 & 72.8 / 58.8 / 27.9 \\
& 70\% & 93.0 & 81.6 / 62.7 / 24.8 & 90.9 & 74.6 / 64.5 / 43.7 & 72.8 / 58.5 / 28.2 \\
& 90\% & 92.3 & 79.8 / 59.5 / 22.2 & 89.5 & 71.6 / 61.0 / 41.1 & 69.9 / 54.7 / 26.1 \\
& 95\% & 90.9 & 77.7 / 57.2 / 20.6 & 86.9 & 66.0 / 54.9 / 37.0 & 64.3 / 49.3 / 23.7 \\
& 99\% & 84.8 & 68.5 / 47.0 / 14.4 & 69.1 & 40.2 / 33.2 / 23.2 & 38.5 / 29.7 / 15.3 \\
\midrule
\parbox[t]{2.5cm}{\multirow{4}{*}[0pt]{\parbox[t]{2.5cm}{\centering Sparse NN\\(Non-Uniform)}}} & 50\% & 93.8 & 83.2 / 64.9 / 27.0 & 89.0 & 73.3 / 62.2 / 43.0 & 71.4 / 55.8 / 25.1 \\
& 90\% & 93.4 & 82.6 / 63.7 / 25.4 & 89.6 & 73.2 / 61.0 / 41.5 & 71.1 / 54.2 / 23.7 \\
& 95\% & 93.7 & 82.8 / 63.4 / 25.8 & 88.1 & 69.9 / 58.3 / 38.9 & 68.1 / 52.5 / 23.9 \\
& 99\% & 92.2 & 79.8 / 60.3 / 21.8 & 85.2 & 62.5 / 52.9 / 35.0 & 61.0 / 48.5 / 22.9 \\
\toprule
\end{tabular}}
\end{table*}
\begin{table*}[p]
\centering
\caption{Detailed evaluations of robust and sparse WRN-28-4 models on CIFAR-100, for different robust ANN pruning approaches and sparsity levels. Robust SNNs are obtained with our conversion method using the corresponding baseline ANNs.}
\label{tab:detail_cifar100}
\scalebox{0.85}{
\begin{tabular}{c c c c c c c c c c c}
\toprule
& \multirow{2}{*}[-2pt]{Sparsity} & \multicolumn{2}{c}{Robust ANN} & \multicolumn{3}{c}{\textbf{Robust SNN (Ours)}} \\
\cmidrule(lr){3-4}\cmidrule(lr){5-7}
& & Clean & PGD & Clean & FGSM$_\text{ens}$ & PGD$_\text{ens}$ \\
\midrule
\parbox[t]{2.5cm}{{\parbox[t]{2.5cm}{\centering Dense NN}}} & 0\% & 67.5 & 46.1 / 27.7 / 8.20 & 69.6 & 43.7 / 30.1 / 14.6 & 40.8 / 23.3 / 5.50 \\
\midrule
\parbox[t]{2.5cm}{\multirow{5}{*}[0pt]{\parbox[t]{2.5cm}{\centering Sparse NN\\(LWM)}}} & 10\% & 67.1 & 46.2 / 28.1 / 8.80 & 69.5 & 44.0 / 30.1 / 14.8 & 40.8 / 23.1 / 5.60 \\
& 30\% & 67.3 & 46.2 / 27.9 / 8.60 & 69.2 & 43.8 / 30.0 / 14.9 & 40.7 / 22.8 / 5.70 \\
& 50\% & 66.9 & 46.0 / 27.5 / 8.10 & 68.9 & 43.3 / 30.1 / 15.2 & 39.9 / 23.2 / 6.00 \\
& 70\% & 66.6 & 44.8 / 26.4 / 7.90 & 67.9 & 41.9 / 28.9 / 14.2 & 38.6 / 21.6 / 5.10 \\
& 90\% & 65.3 & 43.5 / 25.6 / 7.50 & 56.6 & 32.5 / 23.2 / 12.3 & 30.0 / 17.6 / 4.30 \\
\midrule
\parbox[t]{2.5cm}{\multirow{5}{*}[0pt]{\parbox[t]{2.5cm}{\centering Sparse NN\\(Uniform)}}} & 10\% & 67.5 & 46.3 / 27.9 / 8.70 & 69.6 & 43.6 / 30.2 / 14.5 & 40.6 / 23.0 / 5.70 \\
& 30\% & 66.9 & 46.3 / 27.9 / 8.40 & 69.2 & 43.9 / 30.2 / 15.1 & 40.7 / 23.4 / 6.00 \\
& 50\% & 66.8 & 46.0 / 27.5 / 8.10 & 69.1 & 43.6 / 30.2 / 15.0 & 40.5 / 22.8 / 5.90 \\
& 70\% & 67.2 & 45.9 / 27.3 / 8.20 & 68.0 & 43.0 / 30.3 / 15.3 & 39.9 / 23.4 / 6.00 \\
& 90\% & 66.8 & 46.2 / 27.7 / 8.20 & 63.8 & 38.3 / 27.5 / 14.6 & 35.0 / 20.7 / 5.60 \\
\midrule
\parbox[t]{2.5cm}{\multirow{4}{*}[0pt]{\parbox[t]{2.5cm}{\centering Sparse NN\\(Non-Uniform)}}} & 50\% & 67.1 & 46.0 / 27.3 / 8.20 & 68.9 & 43.0 / 30.4 / 14.8 & 40.2 / 23.1 / 6.20 \\
& 90\% & 66.3 & 45.8 / 27.6 / 7.90 & 66.1 & 40.7 / 27.9 / 14.2 & 37.2 / 21.1 / 5.40 \\
& 95\% & 66.8 & 46.6 / 28.4 / 7.8 & 62.9 & 36.9 / 25.0 / 12.0 & 33.2 / 18.0 / 3.70 \\
& 99\% & 60.7 & 40.5 / 22.9 / 5.70 & 44.3 & 23.7 / 16.9 / 8.10 & 21.5 / 11.5 / 2.30 \\
\toprule
\end{tabular}}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\textbf{Varying the PGD$_\text{ens}$ Number of Iterations:}
We investigate the resilience of our SNNs against PGD$_\text{ens}$ attacks with increasing number of iterations, in Table~\ref{tab:pgd_iterations}.
Our main goal is to verify the reliability of gradient approximation with our white-box ensemble attacks.
This analysis is commonly performed as a sanity check in robustness evaluations~\cite{athalye2018obfuscated,carlini2019evaluating}, since our main results only included 10-step PGD$_\text{ens}$.
We observe that in all models PGD$_\text{ens}$ tends to get slightly more effective, hence demonstrating reliable behavior without signs of misleading obfuscated gradients confounding our evaluations, e.g., robust accuracy at 30\% sparsity with PGD$_\text{ens}$ at $\epsilon=8/255$, 7-steps: 28.9\%, 10-steps: 27.9\%, 20-steps: 26.2\%, 40-steps: 25.4\%.
\begin{table*}[t]
\centering
\caption{Evaluations of robust VGG-16 SNNs on CIFAR-10, with increasing number of ensemble PGD attack iterations.}
\label{tab:pgd_iterations}
\scalebox{0.85}{
\begin{tabular}{l cc ccc}
\toprule
& Clean & PGD$_\text{ens}$ (7-steps) & PGD$_\text{ens}$ (10-steps) & PGD$_\text{ens}$ (20-steps) & PGD$_\text{ens}$ (40-steps) \\
\midrule
Dense SNN (0\%) & 92.0 & 75.1 / 60.6 / 29.0 & 74.9 / 59.8 / 27.9 & 74.5 / 58.8 / 26.9 & 74.0 / 58.3 / 25.8 \\
\midrule
Sparse SNN (30\%) & 90.8 & 73.1 / 58.5 / 28.9 & 72.5 / 57.9 / 27.9 & 72.3 / 57.1 / 26.2 & 72.3 / 56.7 / 25.4 \\
Sparse SNN (50\%) & 90.7 & 73.3 / 59.2 / 28.9 & 72.8 / 58.8 / 27.9 & 72.7 / 58.1 / 26.5 & 72.4 / 57.4 / 25.8 \\
Sparse SNN (70\%) & 90.9 & 73.2 / 59.2 / 29.5 & 72.8 / 58.5 / 28.2 & 72.3 / 57.7 / 26.8 & 72.4 / 57.4 / 25.9 \\
Sparse SNN (90\%) & 89.5 & 70.3 / 55.5 / 27.2 & 69.9 / 54.7 / 26.1 & 69.3 / 54.4 / 24.8 & 69.4 / 53.8 / 24.3 \\
\bottomrule
\end{tabular}}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Influence of Adversarial ANN Pretraining on Sparse SNN Robustness}
\label{appendix_annpt}
We demonstrate the impact of using heavier adversarial ANN pretraining on the resulting converted SNN, in terms of the transferred robustness gains.
In Table~\ref{tab:ann_eps} we present evaluations of converted SNNs that use different configurations when pretraining the baseline ANNs.
The robust and sparse finetuning configuration remains the same with $\beta=2$, and $\epsilon=2/255$.
We show that SNNs obtained via conversion are able to leverage and maintain the robustness properties of the baseline ANNs through our robust weight initialization approach.
A higher $\epsilon$ during ANN pretraining also proportionally yields SNNs with higher adversarial robustness as seen in Table~\ref{tab:ann_eps}, although we observe a drop in benign accuracy with increasing $\epsilon$ as expected through the robustness-accuracy trade-off.
Specifically, we could obtain 90\% sparse (uniform) SNNs with clean/robust acc. up to 83.2/40.0, when we perform ANN AT using $\epsilon=8$ and $\lambda_{\text{TRADES}}=6$, as opposed to our previous 89.5/26.1.
\begin{table*}[t]
\centering
\caption{Influence of heavier adversarial ANN pretraining, on the resulting converted SNN. We explore the use of a higher TRADES loss regularization strength and larger perturbation $\epsilon$ values, using VGG-16 with 90\% sparsity on CIFAR-10.}
\label{tab:ann_eps}
\scalebox{0.85}{
\begin{tabular}{l c c c c}
\toprule
& \multicolumn{4}{c}{\centering Adversarial ANN Pretraining with TRADES} \\
\cmidrule(lr){2-5}
& \multicolumn{1}{c}{$\lambda_{\text{TRADES}}=2$} &
\multicolumn{3}{c}{$\lambda_{\text{TRADES}}=6$} \\
\cmidrule(lr){2-2}\cmidrule(lr){3-5}
& $\epsilon=2/255$ & $\epsilon=2/255$ & $\epsilon=4/255$ & $\epsilon=8/255$ \\
\midrule
Clean Acc. & 89.5 & 89.6 & 87.2 & 83.2 \\
\midrule
FGSM$_\text{ens}$ & 71.6 / 61.0 / 41.1 & 72.6 / 62.6 / 44.2 & 70.4 / 62.8 / 47.9 & 65.7 / 59.3 / 46.8 \\
PGD$_\text{ens}$ & 69.9 / 54.7 / 26.1 & 70.9 / 58.0 / 31.9 & 69.2 / 59.7 / 38.3 & 64.8 / 57.3 / 40.0 \\
\bottomrule
\end{tabular}}
\end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Analysis of Layerwise Spike Rates}
\label{sec:appendix_spikerates}
In Figure~\ref{fig:spiking_activity}, we investigate the per-layer spiking rates of 90\% sparse SNNs, to elaborate the emerging differences in energy consumption estimates of the two pruning approaches.
Note that in this particular setting, our estimates revealed that layerwise uniform sparsity yields 6.12$\times$ more, and non-uniform sparsity yields 2.15$\times$ more energy-efficient models than the densely connected SNN.
For layerwise non-uniformly sparse models in Figure~\ref{fig:spiking_activity}(b), we can observe higher spike rates in the layers with higher connectivity.
Since layers with higher connectivity contain more outgoing connections where spikes are transmitted for AC operations, more energy is consumed overall.
Naturally, this effect can occur when we use pruning methods that optimize the connectivity on a per-layer basis.
In contrast, models obtained by layerwise uniform pruning in Figure~\ref{fig:spiking_activity}(a) show more variant layerwise spike rates.
However, this model appears to be less prone to consume significantly more energy in certain layers, since sparse connectivity is uniformly distributed overall.
\begin{figure*}[t!]
\centering
\subfloat[Layerwise uniform sparsity]{
\includegraphics[width=0.49\textwidth]{spiking_activity_hydra.pdf}}\hfill
\subfloat[Layerwise non-uniform sparsity]{
\includegraphics[width=0.49\textwidth]{spiking_activity_harp.pdf}}
\caption{Comparisons of per-layer average spiking rates of the resulting SNNs with the two pruning approaches. Both models maintain 90\% global sparsity (i.e., 10\% connectivity) and trained on CIFAR-10 using the VGG-16 architecture.}
\label{fig:spiking_activity}
\end{figure*}
\end{document}