Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition
Extracted source text · arXiv 2405.03712
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2024
% ready for submission
% \usepackage{neurips_2024}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2024}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2024}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2024}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{amssymb}
\usepackage{wrapfig}
\usepackage{subcaption}
\usepackage{amsmath}
\title{Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Xiaoyan Su\thanks{Joint first authors.} \ \thanks{Corresponding authors.} \\
Shenzhen University\\
\texttt{2021022023@email.szu.edu.cn} \\
\And
Yinghao Zhu\footnotemark[1] \ \footnotemark[2] \\
Alibaba Digital Media \& Entertainment Group \\
\texttt{22160083@zju.edu.cn} \\
\And
Run Li \\
Lanzhou University of Technology \\
\texttt{lirun4977@gmail.com} \\
}
\begin{document}
\maketitle
\begin{abstract}
In the past, research on a single low dimensional activation function in networks has led to internal covariate shift and gradient deviation problems. A relatively small research area is how to use function combinations to provide property completion for a single activation function application. We propose a network adversarial method to address the aforementioned challenges. This is the first method to use different activation functions in a network. Based on the existing activation functions in the current network, an adversarial function with opposite derivative image properties is constructed, and the two are alternately used as activation functions for different network layers. For complex situations, we propose a method of high-dimensional function graph decomposition(HD-FGD), which divides it into different parts and then passes through a linear layer. After integrating the inverse of the partial derivatives of each decomposed term, we obtain its adversarial function by referring to the computational rules of the decomposition process. The use of network adversarial methods or the use of HD-FGD alone can effectively replace the traditional MLP+activation function mode. Through the above methods, we have achieved a substantial improvement over standard activation functions regarding both training efficiency and predictive accuracy. The article addresses the adversarial issues associated with several prevalent activation functions, presenting alternatives that can be seamlessly integrated into existing models without any adverse effects. We will release the code as open source after the conference review process is completed.
\end{abstract}
\section{Introduction}
With the explosion of LLM, deep learning has once again become an important research topic in many scientific fields. Both LLM and various neural network models rely on the nonlinearity introduced by activation functions, and using appropriate activation functions can represent and learn more complex functional relationships. In order to enhance the nonlinear expression ability of the model, many non parametric activation functions that are suitable for specific tasks have been proposed according to the different application tasks. The currently popular activation functions include Sigmoid, Tanh, ReLu, GeLu \cite{hendrycks2016gaussian}, Swish \cite{ramachandran2017searching}, etc. Regardless of the activation function, it is limited to introducing nonlinearity in a single two-dimensional space, inevitably causing changes in the distribution of layer inputs, leading to internal covariate shift (ICS) \cite{ioffe2015batch} and gradient deviation problems.
In order to alleviate ICS as much as possible during model training, only the same activation function is considered at different network layers to increase model nonlinearity. As the network depth increases, the gradient may have varying degrees of deviation, which leads to ineffective training \cite{dubey2022activation}. Therefore, we consider whether it is possible to use an activation function to generate ICS in the model, while also having a non-linear adversarial function to correct the gradient, pulling the trained data distribution back to the original data distribution, so that the gradient can be smoothly transmitted back.
\textbf{Our Contributions.} Given the challenges posed by introducing activation functions in the network, this paper proposes a network adversarial method. For complex activation functions, the adversarial function is solved through high-dimensional function graph decomposition (HD-FGD). The partial derivative of the function is derived from the derivative of the original activation function. Although the image properties are opposite, it does not introduce additional nonlinearity to the model and cause overfitting. By cross using the original activation function and its adversarial function in the network layer of the model to activate the network, gradient updates are made smooth during backpropagation. Our main contributions are in three aspects as follows
\begin{itemize}
\item Propose a universal network adversarial method and incorporate it into the model system, which can effectively solve the problems of ICS and gradient deviation during the training process, while significantly improving the predictive performance of the model.
\item For the solution of adversarial functions for complex activation functions, we propose a method based on HD-FGD to simplify the process. Users split it into different expressions according to their wishes. After verification, the split activation function performs significantly better than the original activation function, providing a new idea for the transformation of two-dimensional activation functions into high-dimensional research.
\item The effectiveness of network adversarial method has been demonstrated, and the adversarial functions of common activation functions have been solved. The combination of the corresponding activation functions and their adversarial functions can replace the corresponding activation functions in all linear layers of the model without loss, and significantly accelerate the convergence speed of the model.
\end{itemize}
For tasks in different fields, our method was validated in the network layers of multiple models. Experiments have shown that compared to the original activation function, using network adversarial in the network layers of each model can significantly improve model performance and training speed. The activation function obtained by using HD-FGD can effectively replace the original activation function, especially in the linearity layer, with significant effects.
\section{Related Work}
\subsection{The Evolution of Nonlinear Learning}
Early neural networks introduced nonlinearity through non parametric activation functions such as Sigmoid and ReLu \cite{eger2019time, liu2019natural, basirat2020relu, gu2019fast}. With the widespread application of convolutional networks, new activation functions such as ELU \cite{clevert2020fast, trottier2017parametric} aim to achieve higher performance cross structures in models. Although the effect is significant, it relies more on architecture and task attributes. In order to achieve adjustable nonlinear expression of the model during training, an adaptive activation function APL \cite{agostinelli2014learning} with learnable parameters is proposed, which performs excellently in multi-scale tasks. At the same time, there are also studies attempting to make the activation function learnable to adapt to different task attributes. For example, Zoph et al. \cite{zoph2016neural} used reinforcement learning to train multiple networks and generated nonlinearity in the results. Although this method generated more generalized activation functions, it cannot guarantee the existence of the optimal generalizable function in strict space. Hyperactivations \cite{vercellino2017hyperactivations} innovatively explores the search space of activation functions and uses hypernetworks to generate weights for the activation network, effectively improving network training speed and predictive performance.
Overall, in the past, most activation functions with different properties were suitable for certain tasks in specific fields. Nowadays, the nonlinear design of activation functions tends to be biased towards mining potential complex relationships in specific task data, especially Hyperactivations \cite{vercellino2017hyperactivations}, which uses a nested activation network with generative weight networks, providing a new paradigm for improving the expression ability of the required attributes of activation functions.
\subsection{ICS and Gradient Deviation}
The promotion of deep learning benefits from the progress of many algorithms and architectures, such as Dropout \cite{srivastava2014dropout}, initial weight initialization methods \cite{sutskever2013importance, he2016identity, klambauer2017self, he2015delving}, optimizers \cite{diederik2014adam, zhuang2020adabelief, luo2019adaptive}, etc. Among them, the most prominent example is batch normalization \cite{ioffe2015batch}, which improves neural networks by introducing additional network layers to control the mean and variance of layer distributions, and can solve the problem of ICS from the dimension of data distribution correction. Based on this, the layer normalization method \cite{ba2016layer} has been proposed, which is not affected by batch size and is more suitable for processing sequential data tasks such as natural language processing. In addition, there is also group normalization technique \cite{wu2018group}, which divides each sample feature into multiple groups and normalizes them within each group, reducing dependence on global information and memory overhead. Other normalization based algorithms \cite{li2018adaptive, shi2016real} have also been proposed. SELU \cite{klambauer2017self} uses a weight initialization method based on linear transformation to improve the training stability of the network, but its hyperparameters are affected by human intervention and have higher requirements for the distribution of input data, making it more suitable for specific fields. In Mattia J. Villani's work \cite{jiang2022delve}, it was demonstrated that deep neural networks obtained through the ReLU function exhibit a certain degree of degradation. Specific methods can be used to express deep networks using shallow networks, further simplifying the network structure and reducing the possibility of gradient shift.
In summary, the above contributions have paved the way for the research of deep learning in various fields and provided new ideas for the subsequent ICS and gradient shift problems, such as enabling the activation function to have adaptive mechanisms during training.
\section{Network Adversarial Method}
In order to better stabilize the gradient improvement model prediction ability during the training process and minimize ICS, we propose a network adversarial method, which is divided into global adversarial (GA) and split adversarial (SA). The HD-FGD in SA effectively replaces the original activation function, while proposing the idea of mapping two-dimensional activation functions to high-dimensional space, providing new exploration space for the research of deep learning in this part.
\subsection{Verification of Gradient Adversarial Thinking}
Assuming a k-layer deep neural network is used for classification tasks, and all linear layer activation functions are Sigmoid, $F(x)$ represents its derivative function, $x$ is the feature data of the input model, $y$ is the true label, $\hat y$ is the model prediction result, $w_i$ is the parameter of the i-th layer, $x_i$ is the input data of the i-th layer, $l$ is the loss function, and the gradient $\lambda_i$ of each layer can be expressed as
\begin{equation}\label{f1}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
\lambda _i = \frac{{\partial l}}{{\partial w_i }} = \frac{{\partial l}}{{\partial \hat y}} \cdot \frac{{\partial \hat y}}{{\partial w_i }} \\
= \frac{{\partial l}}{{\partial \hat y}} \cdot \frac{{\partial (x_k w_k )}}{{\partial x_k }} \cdot \frac{{\partial (x_{k - 1} w_{k - 1} )}}{{\partial x_{k - 1} }} \cdot \frac{{\partial (x_{k - 2} w_{k - 2} )}}{{\partial x_{k - 2} }} \ldots \frac{{\partial (x_1 w_1 )}}{{\partial w_1 }} \\
= \frac{{\partial l}}{{\partial \hat y}} \cdot w_k F(x_k ) \cdot w_{k - 1} F(x_{k - 1} ) \cdot w_{k - 2} F(x_{k - 2} ) \ldots x_i \\
= \frac{{\partial l}}{{\partial \hat y}} \cdot (\prod\limits_{j = 1}^{k - i} {w_{k - j + 1} \cdot F(x_{k - j + 1} )} ) \cdot x_i \\
\end{array}
\end{equation}
The gradient of each layer is influenced by two parts: the multiplication between the parameters of each previous layer and the gradient of the input data for each layer, and the other part is the input data for that layer. In the work of Sergey et al. \cite{ioffe2015batch}, the use of batch normalization effectively controls the stability of the input distribution in the control layer, and we assume that it approximates a normal distribution, The derivative interval of Sigmoid is between 0 and 0.25. As the network deepens, excessive multiplication will cause the gradient to disappear, while multiplication with a derivative interval greater than 1 will cause the gradient to explode. For formula (\ref{f1}), some of the formulas can be split into pairwise groups.
\begin{equation}\label{f2}
\lambda _i = \frac{{\partial l}}{{\partial \hat y}} \cdot (\prod\limits_{j = 1}^{(k - i)/2} {w_{k - (2j - 1) + 1} \cdot \underbrace {F(x_{k - (2j - 1) + 1} )}_A} \cdot w_{k - (2j - 1)} \cdot \underbrace {F(x_{k - (2j - 1)} )}_B) \cdot x_i
\end{equation}
The gradient stability interval of a single activation function is limited. If $A$ and $B$ in formula (\ref{f2}) can generate gradient pulling and have gradient stability intervals respectively, there is a greater possibility that gradient deviation will not occur during model training. The two are alternately used as activation functions for different network layers during backpropagation, forming gradient adversaries to maintain a stable state of the gradient. Compared with the study of a single activation function, we study the combination of two or more activation functions to flexibly control the required properties of the activation function, reduce the frequency of gradient deviation and the amplitude of ICS. This is the main idea of network adversarial method.
To validate our idea, the following experiment is proposed. We construct a multi-layer linear network and add batchnorm layers while using Sigmoid activation as the control group. The experimental group, on the basis of the above, uses Sigmoid and its adversarial function $\xi _{{\rm{Sigmoid}}} (x)$ alternately as the activation function. The following image shows the data distribution, gradient distribution, and predictive performance after passing through different linear layers, and these results further validate the feasibility of our idea.
\begin{figure}
\centering
\begin{minipage}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{1_a.png}
\subcaption{(a) Gradient distribution}
% \label{fig:1-a}
\end{minipage}
\hfill
\begin{minipage}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{1_b.png}
\subcaption{(b) Data distribution}
% \label{fig:1-b}
\end{minipage}
\hfill
\begin{minipage}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{1_c.png}
\subcaption{(c) Predictive performance}
% \label{fig:1-c}
\end{minipage}
\caption{Measure ICS and gradient deviation in networks with and without network adversarial methods. Sigmoid-Adv in the figure is $\xi _{{\rm{Sigmoid}}} (x)$. According to the changes in gradient distribution (a), it can be seen that the linear layer gradient distribution with the addition of network adversarial method is more uniform, and the distribution of extreme gradients is significantly improved. The data distribution of each linear layer in the experimental group (b) fluctuates significantly less than that in the control group. In terms of performance (c), the MSE of the experimental group maintains a significant advantage in all iterations, and there is no significant fluctuation.}
\label{fig:1}
\end{figure}
\subsection{The Proposal of HD-FGD}
We will provide a general explanation of the network adversarial method proposed in this article. Specifically, for simple activation functions such as Sigmoid, we can directly perform GA operations. For complex activation functions such as Tanh and GeLu, we need to use SA operations. Otherwise, there may be difficulties in solving or exponential growth of adversarial gradients. Taking Tanh as an example, the adversarial function \(\frac{{e^{4x} - 1}}{{8e^{2x} }} + \frac{x}{2} + C\) is solved using GA. During the backpropagation process, the adversarial gradient is more prone to gradient deviation due to the risk of exponential explosion, leading to training termination. The SA based on HD-FGD is to represent complex functions with multiple simple functions, and provide a trainable network for different splitting terms to achieve the transformation from low dimensional functions to high dimensions, simplifying the solving process while ensuring stable training.
\begin{figure}
\centering
\includegraphics[width=0.94\textwidth]{2.pdf}
\caption{Schematic diagram of network adversarial method.}
\label{fig:2}
\end{figure}
As shown in Figure \ref{fig:2}, network adversarial method can be divided into two forms: GA and SA. For activation function $f(x)$, when using GA, the reciprocal of its derivative is the derivative $\xi(x)^{'}$ of adversarial function $\xi(x)$. After integrating $\xi(x)^{'}$, $\xi(x)$ is obtained.
For the scaling mapping $\chi = AF\left({Wx}\right)$ of a single linear layer of data input, this process can essentially be seen as adding a trainable matrix $W$ to any activation function $f(x)$, which can be represented as $f(W,x)$, where $AF\left( \cdot \right)$ represents the activation function. Therefore, it can be considered to replace this linear mapping with multiple linear mappings $f(\chi _1 ,\chi _2 \ldots \chi _n )$, each of which introduces nonlinearity with different properties to the model, making the expression of the activation function more flexible. This splitting can reduce the number of hidden units h in the original trainable parameter matrix to $n$ trainable parameter matrices with $h/n$ hidden units, and optimize the computing speed of $W_i$ using parallel operations.As shown in Figure \ref{fig:3}, under normal circumstances, the non-linear introduction of the model is represented by $x \cdot W \to f(x) \to x'$, where $x'$ represents the output after linear mapping and activation function, while HD-FGD optimizes the process of $W \to f(x)$, while transforming two-dimensional functions into high-dimensional space.
When using SA, first convert the original activation function into a linearly mapped expression $f(x) = f(W,x)$, the original activation function is decomposed into a computational structure composed of basic operations using HD-FGD. In the simplest case, if all basic operation symbols are the same, the original activation function can be represented as the accumulation $\sum\limits_{i = 1}^n {W_i g_i (x)}$ or multiplication term of each splitting term $\prod\limits_{i = 1}^n {W_i g_i (x)}$, where $g_i(x)$ is the different parts of the original activation function after splitting. For the split $g_i(x)$, set their respective trainable parameter matrices $W_i$, where $n$ represents the specific number of split terms, which is determined by the specific expression of the activation function or can be artificially split into a specified value. When splitting, it is best to use methods such as \cite{ioffe2015batch} to process the input data as a zero mean, while ensuring that there is a saturation interval in the value range of $g_i(x)$. Otherwise, the solved adversarial function needs to be truncated using clip \cite{bungert2021clip} or clamp \cite{pascanu2013difficulty}. Following the above requirements can accelerate network convergence and reduce the risk of gradient deviation. According to $\chi = AF\left({Wx}\right)$, there is $f(x) = \sum\limits_{i = 1}^n {\varphi (\chi _i )}$, where $\varphi (\chi _i )$ represents the i-th splitting term after passing through the linear layer, and the sum of partial derivatives of $f(x)$ can be expressed as $\sum\limits_{i = 1}^{\rm{n}} {\gamma _i \cdot \chi _i }$, where $\gamma _i$ is a constant term. The sum of partial derivatives of $\xi (x)$ can be expressed as $\xi (\chi _1 ,\chi _1 \ldots \chi _{\rm{n}} )^{'} = \sum\limits_{i = 1}^{\rm{n}} {\frac{1}{{\gamma _i \chi _i }}} $, and by integrating it, the adversarial function $\xi (x) = \xi (\chi _1 ,\chi _2 \ldots \chi _{\rm{n}} ) = \sum\limits_{i = 1}^{\rm{n}} {\int {\frac{1}{{\gamma _i \chi _i }}} } {\rm{d}}\chi _i$ can be obtained. When using the adversarial function as the activation function, in order to avoid overfitting caused by too many splitting terms, we add an L2 penalty term in gradient backpropagation as a constraint.
Taking Tanh as an example in Figure \ref{fig:2}, it can be transformed into an expression for Sigmoid
\begin{equation}\label{f3}
{\rm{Tanh}}(x) = \frac{{e^x - e^{ - x} }}{{e^x + e^{ - x} }} = \frac{{2e^x - e^x - e^{ - x} }}{{e^x + e^{ - x} }} = \frac{2}{{1 + e^{ - 2x} }} - 1 = 2{\rm{Sigmoid}}(2x) - 1
\end{equation}
Using a SA based on HD-FGD to solve its adversarial function, it can be divided into two parts, each of which passes through a linear mapping network $W_i$. Let $\chi _i = W_i x$, then Tanh can be expressed as
\begin{equation}\label{f4}
{\rm{Tanh(}}x{\rm{)}} = {\rm{Tanh}}(\chi _1 ,\chi _2 ) = {\rm{Sigmoid}}(\chi _1 ) + {\rm{Sigmoid}}(\chi _2 ) - 1
\end{equation}
At this point, the operation realizes the mapping of two-dimensional Tanh to high-dimensional space, so the study of Tanh's derivative can also shift from two-dimensional to three-dimensional space. The derivative of its adversarial function is the sum of the reciprocal of the partial derivatives of ${\rm{Tanh}}(\chi _1 ,\chi _2 )$, and after integration, it becomes Tanh's adversarial function.
Based on experimental verification, we have gained some experiences that can make network adversarial method more effective:
\textbf{Small gradients are effective in the end.} For any neural network model, we alternately use the original activation function and its adversarial function as the activation function for the corresponding layer in the linear layer. In backpropagation, deep network gradients affect shallow network gradients, so the activation function gradient in the latter layer can counteract the gradient in the previous layer, but the latter layer can only rely on deeper gradients to counteract it. Taking Sigmoid as an example, if the activation function of the last layer of the network is its adversarial function, its gradient will not have any small gradients against it, which is prone to gradient explosion. Therefore, the activation function of the latter layer of the network should be as conservative as possible, and choosing a small gradient activation function in the latter layer will improve the performance of the network.
\textbf{Better results are achieved when used in conjunction with batch normalization.} When batch normalization is applied to the input network data, most gradients are well constrained within the stable range of the activation function. For Sigmoid, it avoids gradient disappearance caused by extreme inputs, and the derivative symmetric activation function can better handle extreme inputs.
\textbf{The activation function's derivative being symmetric about the y-axis leads to better performance.} If the gradient at $x_0$ of a certain network layer is $f(x_0 )^{'}$, the presence of $\xi (x)^{'} $ in $x \in [x_0 - \tau ,x_0 + \tau ]$ in the next network layer of backpropagation can pull $f(x_0 )'$ to a stable interval, and $\tau $ represents the allowable deviation range. At this time, $x \in [ x_0 - \tau, x_0 + \tau]$ is called the stable interval of the function combination of $x$ at $x_0$. At this point, if $f(x)'$ is symmetric about the y-axis, due to the opposite nature of the image between $f(x)'$ and $\xi (x)^{'} $, $\xi (x)^{'} $ is also symmetric about the y-axis. Furthermore, there exists a situation where $x \in [ - (x_0 - \tau ), - (x_0 + \tau )]$ also pulls $f(x)'$ back to the stable interval. Therefore, compared to asymmetric functions, the stable interval of the function combination is doubled.
\section{Application Instance}
This section takes Sigmoid as an example and uses GA to solve its adversarial function as a whole. Taking Tanh and GeLu (refer to the Appendix \ref{A-2-2} for the derivation process) as examples, it shows the process of using HD-FGD to separate them into four terms and using SA to solve their adversarial functions separately.
\subsection{The Adversarial Function of Sigmoid} \label{4-1}
Using the GA of the network adversarial method to solve the adversarial function of Sigmoid, the derivation process is as follows
\begin{equation}\label{f5}
{\rm{Sigmoid}}(x) = \frac{{\rm{1}}}{{{\rm{1}} + e^{ - x} }} \Rightarrow {\rm{Sigmoid}}(x)^{'} = \frac{{e^x }}{{{\rm{(1}} + e^x )^2 }} \Rightarrow \xi _{{\rm{Sigmoid}}} (x)^{'} = \frac{{{\rm{(1}} + e^x )^2 }}{{e^x }}
\end{equation}
Among them, $\xi _{{\rm{Sigmoid}}} (x)^{'}$ represents the derivative of the Sigmoid adversarial function, which is integrated to obtain the adversarial function $\xi _{{\rm{Sigmoid}}} (x)$. Using the substitution method, define $u = e^{x} $, then ${\rm{d}}u = e^{x} {\rm{d}}x$, and the integration of $\xi _{{\rm{Sigmoid}}} (x)^{'}$ (derivation can be found in Appendix \ref{A-1-1}) is as follows
\begin{equation}\label{f6}
\int {\frac{{{\rm{(1}} + e^x )^2 }}{{e^x }}} {\rm{d}}x = \frac{{e^{2x} - 1}}{{e^x }} + 2x + C
\end{equation}
In deep network training, even if ReLu grows at a rate of $y=x$, gradient deviation may occur. For Sigmoid and its adversarial functions, gradient deviation is mainly caused by the uncontrollability of network layer parameters under extreme input conditions. To address this issue, the derivative of the Sigmoid can be raised by $\alpha$ units on the original basis, with its derivative becoming $\frac{{e^x }}{{(1 + e^x )^2 }} + \alpha $ and the value interval of the derivative becoming $\left( {\alpha ,\alpha + 0.25} \right)$. At this point, the derivative of the adversarial function is reciprocal to the function after the Sigmoid derivative is raised, and its derivative becomes $\frac{{(1 + e^x )^2 }}{{e^x + \alpha (1 + e^x )^2 }}$. Therefore, the value interval of the derivative of the adversarial function becomes $\left(\frac{1}{{0.25 + \alpha }})\right)$, $\frac{1}{\alpha }$. During training, the parameter $\alpha$ is set as an adjustable parameter, and in this experiment, it is set to 1. When using the Sigmoid function and its adversarial function alternately, the gradient shift generated in a certain layer can be appropriately scaled by the value range of the adversarial function in the next layer, thereby correcting the shift. Due to the limitation of $\alpha$, gradient deviation is avoided in the gradient correction of the derivative of the adversarial function. The modified Sigmoid function ${\rm{sigmoid}}_\vartheta $ can be found in formula (\ref{f7}), and its adversarial function $\xi _{{\rm{sigmoid}}_\vartheta } (x)$ (solution process can be found in Appendix \ref{A-1-2}) can be found in formula (\ref{f8}).
\begin{equation}\label{f7}
{\rm{Sigmoid}}_\vartheta (x) = \frac{1}{{1 + e^{ - x} }} + \alpha x + C
\end{equation}
\begin{equation}\label{f8}
\xi _{{\rm{Sigmoid}}_\vartheta } (x) = \frac{{{\rm{ln}}(2\alpha e^x + \sqrt {4\alpha + 1} + 2\alpha + 1) - {\rm{ln}}(\left| {2\alpha e^x - \sqrt {4\alpha + 1} + 2\alpha + 1} \right|)}}{{\alpha \sqrt {4\alpha + 1} }} + \frac{x}{\alpha } + C
\end{equation}
Among them, $\alpha$ in formula (\ref{f8}) is greater than 0.
\subsection{The Adversarial Function of Tanh}
The expression of Tanh is $\frac{{e^x - e^{ - x} }}{{e^x + e^{ - x} }}$, and its adversarial function is solved using the SA in the network
\begin{wrapfigure}{r}{0.5\textwidth}
\centering
\includegraphics[width=0.5\textwidth]{3.pdf}
\caption{HD-FGD splits Tanh into four terms.}
\label{fig:3}
\end{wrapfigure}
adversarial method. Due to differences in expression or the number of split terms, the split form of the same function does not have uniqueness. In addition to the 2-term splitting method in Figure \ref{fig:2}, Tanh can also be split into 4 terms. Firstly, add a linear layer to the Tanh function, and the network can be expressed as $\frac{{e^{Wx} - e^{ - Wx} }}{{e^{Wx} + e^{ - Wx} }}$, where $W$ represents the trainable parameter matrix. After specifying the number of split terms as $n=4$, a separate trainable parameter matrix $W_i$ can be added for each split part. Instead of learning a complex function relationship, $W$ now learns $n$ simple function relationships. Since each $W_i$ is used to learn the corresponding function relationship, the number of hidden units in $W_i$ can be moderately reduced. We define the number of hidden units for each split part in the linear layer as $\frac{1}{n}$. This process can be illustrated as shown in Figure \ref{fig:3}.
The split Tanh can be expressed as
\begin{equation}\label{f9}
{\rm{Tanh}}\left( {W_1 ,W_2 ,W_3 ,W_4 ,x} \right) = \frac{{e^{W_1 x} - e^{W_2 x} }}{{e^{W_3 x} + e^{W_4 x} }}
\end{equation}
Let $\chi _i = W_i x$, then formula (\ref{f9}) can be modified to
\[
{\rm{Tanh}}\left( {\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 } \right) = \frac{{e^{\chi _1 } - e^{\chi _2 } }}{{e^{\chi _3 } + e^{\chi _4 } }}
\]
According to the network adversarial framework, the partial derivative of ${\rm{Tanh}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$ can be expressed as
\begin{equation}\label{f10}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\left\{ \begin{array}{l}
\frac{{\partial {\rm{Tanh}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _1 }} = \frac{{e^{\chi _1 } }}{{e^{\chi _3 } + e^{ - \chi _4 } }} \\
\frac{{\partial {\rm{Tanh}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _2 }} = \frac{{e^{ - \chi _2 } }}{{e^{\chi _3 } + e^{ - \chi _4 } }} \\
\frac{{\partial {\rm{Tanh}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _3 }} = \frac{{\left( {e^{ - \chi _2 } - e^{\chi _1 } } \right)e^{\chi _3 } }}{{\left( {e^{\chi _3 } + e^{ - \chi _4 } } \right)^2 }} \\
\frac{{\partial {\rm{Tanh}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _4 }} = \frac{{\left( {e^{\chi _1 } - e^{ - \chi _2 } } \right)e^{ - \chi _4 } }}{{\left( {e^{\chi _3 } + e^{ - \chi _4 } } \right)^2 }} \\
\end{array} \right.
\end{equation}
Taking the reciprocal of each partial derivative in formula \ref{f10} is the partial derivative of the adversarial function $\xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$, which can be expressed as follows
\begin{equation}\label{f11}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\left\{ \begin{array}{l}
\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _1 }} = \frac{{e^{\chi _3 } + e^{ - \chi _4 } }}{{e^{\chi _1 } }} \\
\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _2 }} = \frac{{e^{\chi _3 } + e^{ - \chi _4 } }}{{e^{ - \chi _2 } }} \\
\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _3 }} = \frac{{\left( {e^{\chi _3 } + e^{ - \chi _4 } } \right)^2 }}{{\left( {e^{ - \chi _2 } - e^{\chi _1 } } \right)e^{\chi _3 } }} \\
\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _4 }} = \frac{{\left( {e^{\chi _3 } + e^{ - \chi _4 } } \right)^2 }}{{\left( {e^{\chi _2 } - e^{ - \chi _2 } } \right)e^{ - \chi _4 } }} \\
\end{array} \right.
\end{equation}
Set the constant coefficients in each partial derivative as trainable parameter $\alpha_i$, initialize $\alpha_i$ as a unit constant according to the formula (Initialization can be found in Appendix \ref{A-2-1}). Clculate the partial derivatives and integrals in formula (\ref{f11}) according to the operation rules when splitting, the adversarial function of ${\rm{Tanh}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$ is obtained as follows
\begin{equation}\label{f12}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
- \left( {\alpha _1 + \alpha _2 + \alpha _4 + \alpha _5 } \right){\rm{e}}^{ - \chi _1 } + \frac{{\alpha _7 \left( {{\rm{e}}^{ - \chi _3 } \alpha _8 ^{ - 2} - {\rm{e}}^{ - \chi _3 } - 2\alpha _8 ^{ - 1} \chi _3 } \right)}}{{\alpha _7 \alpha _8 - 1}} + \\
\frac{{\alpha _{10} \left( {{\rm{e}}^{\chi _4 } \alpha _2 ^2 - {\rm{e}}^{ - \chi _4 } + 2\alpha _{12} \chi _4 } \right)}}{{\alpha _{10} \alpha _{11} - 1}} + \alpha _3 + \alpha _6 + \alpha _9 + \alpha _{12} \\
\end{array}
\end{equation}
\section{Experiment} \label{5}
In this section, we presented our experimental results on image classification using the CIFAR10 \cite{sharma2018analysis} and CIFAR100 \cite{sharma2018analysis} datasets, demonstrating the effectiveness of our network adversarial method in improving model performance.The specific structures of the VIT, ResNet, and SwT models in this experiment are VIT Base, ResNet34, and Swin Transformer tiny, respectively. We implemented adversarization based on Sigmoid, Tanh, and GeLu activation functions using PyTorch 2.1.2. Additionally, we employed various HD-FGD for Tanh and GeLu. It is worth noting that although the decomposition process for GeLu is more complex, the resulting adversarial function is relatively concise (reference to Appendix \ref{A-2-2}). All experiments were conducted on NVIDIA GeForce RTX 3090 with 24GB memory, and the reported results are averages over 5 random seeds. Furthermore, we applied our method to machine translation tasks in the NLP domain (reference to Appendix \ref{A-4}), where we observed significant improvements in model performance.
\subsection{GA is Effective}
The simplified form of activation functions only requires the GA to achieve gradient correction during training. Taking ${\rm{Sigmoid}}_\vartheta (x)$ in \ref{4-1} as an example, we compared the network architecture performance of VIT, ResNet, and Swin-Transformer (SwT) models on different image classification tasks by alternately using the activation function as either $\xi _\vartheta (x)$ or ${\rm{Sigmoid}}_\vartheta (x)$ in the linear and convolutional layers. Table \ref{table-1} shows that all models consistently outperformed the baseline in terms of GA ACC and Top-5 Error, providing 3.87\% to 1466.11\% ACC and 7.00\% to 82.00\% Top-5 Error performance improvements for different models, respectively. It can be clearly seen from Figure \ref{fig:4} that the loss curve is significantly more stable after using the GA in different task scenarios.
\subsection{HD-FGD is Effective} \label{5-2}
It's worth noting that the HD-FGD not only provides generality for network adversarial training but can also be individually applied to ordinary networks to replace the original activation functions. Tanh and Gelu are both decomposed into four components using the HD-FGD method, and the decomposed activation functions are used in place of the original activation functions during model training in the linear layers. Table \ref{table-2} shows the new activation functions obtained by various models through the HD-FGD method, achieving higher ACC and Top5-Error on different datasets compared to the original activation functions. For more detailed experimental analysis, please refer to Appendix \ref{A-3-2}. Since the HD-FGD uses parallel computation to solve the parameter values of trainable matrices, it can improve the training speed of all models. Furthermore, in \ref{5-3}, it is entirely feasible to replace the original activation functions with the decomposed activation functions obtained through HD-FGD.
\subsection{SA is Effective} \label{5-3}
To further evaluate the effectiveness of SA in network adversarial methods, we based our analysis on the experiments outlined in Section \ref{5-2}. We utilized the activation functions optimized by HD-FGD for Tanh and GeLu as the original activation functions for each model. Subsequently, SA was employed for model training. The results presented in Table \ref{table-3} indicate that applying SA on top of HD-FGD significantly enhances model prediction performance, with a maximum improvement of 104.64\% in ACC. For a more detailed analysis of the experiments, please refer to Appendix \ref{A-3-3}. The effectiveness of SA primarily depends on the complexity of the activation function decomposition. If HD-FGD proves effective, further leveraging SA will become the preferred method for enhancing model performance.
\section{Conclusion}
In this work, we propose a network adversarial method that achieves gradient adversarial training during model training. We demonstrate the experimental results of using this method with various activation functions compared to the current conventions in different domains. The results show that both GA and SA significantly improve model prediction performance. Additionally, HD-FGD accelerates model training speed and contributes to notable performance improvements. Based on our method, we recommend researchers who still use standard activation functions to rewrite their networks at a lower cost. HD-FGD and network adversarial method can potentially bring substantial performance enhancements to the rewritten models.
The network adversarial method innovatively uses a combination of two functions during training as the activation of network layers, alternately correcting the gradients. In the future, it is possible to consider using functions with different properties or more function combinations as the nonlinear activations of the network to achieve smooth adversarialization between gradients.
The introduction of HD-FGD splits the original activation function into multiple terms while also transforming the activation function from a two-dimensional space to a higher-dimensional space. The projection of the spatial activation function onto various planes reflects the mapping of the derivatives of different decomposition terms in the two-dimensional plane, providing feasibility and new ideas for exploring the activation function in higher dimensions.
\textbf{Limitations} \quad Although the network advisory method has achieved exciting results, there are also some remaining issues. Due to the need to customize the number of items and expression of the original activation function for HD-FGD, different definitions will bring different gain effects. In addition, the solution of the adversarial function depends on the specific expression of each splitting term. If the final effect does not reach the ideal state, it is necessary to redefine the splitting form or number of splitting terms of the activation function from the HD-FGD stage. However, as shown in our experiment, our method remains effective even though custom split forms may bring different effects. Although it is necessary to manually derive integrals during the solving of adversarial functions, the solved adversarial functions can be applied to all models.
\bibliographystyle{plain}
\bibliography{ref.bib}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\section{Appendix}
\subsection{Formula Derivation}
\subsubsection{Solving the Adversarial Function of Sigmoid} \label{A-1-1}
By using the method of substitution to solve this integral, let's define $u = e^x $. Then ${\rm{d}}u = e^x {\rm{d}}x$, the integral of $\xi _{{\rm{Sigmoid}}} (x)^{'}$ can be computed as follows
\[
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
\int {\frac{{\left( {{\rm{e}}^x + 1} \right)^2 }}{{{\rm{e}}^x }}} {\rm{d}}x \\
= \int {\frac{2}{u}} + \frac{1}{{u^2 }} + 1{\rm{d}}u \\
= 2\int {\frac{1}{u}} {\rm{d}}u + \int {\frac{1}{{u^2 }}} {\rm{d}}u + \int {1{\rm{d}}u} \\
= 2\ln \left( {\left| u \right|} \right) + \frac{{u^2 - 1}}{u} \\
= \frac{{{\rm{e}}^{2x} - 1}}{{{\rm{e}}^x }} + 2x + C \\
\end{array}
\]
\subsubsection{Improved Sigmoid Adversarial Function Solution} \label{A-1-2}
By using the method of substitution to solve this integral and defining $u = e^x$, we obtain ${\rm{d}}u = e^x {\rm{d}}x$. The integral of $\xi _{{\rm{Sigmoid}}_\vartheta } (x)'$ can be computed as follows
\begin{equation}\label{f13}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
\int {\frac{{\left( {{\rm{e}}^x + 1} \right)^2 }}{{{\rm{e}}^x + \alpha \left( {{\rm{e}}^x + 1} \right)^2 }}} {\rm{d}}x \\
= \int {\frac{{u^2 + 2u + 1}}{{\alpha u^3 + \left( {2\alpha + 1} \right)u^2 + \alpha u}}{\rm{d}}} u \\
= \int {\frac{{u^2 + 2u + 1}}{{u\left( {\alpha u^2 + \left( {2\alpha + 1} \right)u + \alpha } \right)}}{\rm{d}}} u \\
= \int {\frac{{u^2 + 2u + 1}}{{\alpha u\left( {u - \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}} \right)\left( {u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}} \right)}}} {\rm{d}}u \\
\end{array}
\end{equation}
Partial fraction decomposition of formula (\ref{f13})
\begin{equation}\label{f14}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
\int {\frac{{u^2 + 2u + 1}}{{\alpha u\left( {u - \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}} \right)\left( {u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}} \right)}}{\rm{d}}u} \\
= \int {\frac{1}{\alpha } \cdot \left( {\frac{A}{{u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}}} + \frac{B}{{u - \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}}} + \frac{C}{u}} \right)} {\rm{d}}u \\
= \int {\frac{1}{\alpha } \cdot \frac{{\left( {2\alpha C + 2\alpha B + 2\alpha A} \right)u^2 + \left( {\left( {4\alpha + 2} \right)C + \left( {2\alpha + 1} \right)B + \left( {2\alpha + 1} \right)A} \right)u + \sqrt {4\alpha + 1} \left( {B - A} \right)u + 2\alpha C}}{{2u\left( {u - \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}} \right)\left( {u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}} \right)}}} {\rm{d}}u \\
\end{array}
\end{equation}
From formula (\ref{f14}), we can obtain a system of equations based on the coefficients of equal powers
\begin{equation}\label{f15}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\left\{ \begin{array}{l}
u^0 :C = 1 \\
u^1 :\frac{C}{\alpha } + 2C + \frac{{B\sqrt {4\alpha + 1} }}{{2\alpha }} + \frac{B}{{2\alpha }} + B - \frac{{A\sqrt {4\alpha + 1} }}{{2\alpha }} + \frac{A}{{2\alpha }} + A = 2 \\
u^2 :A + B + C = 1 \\
\end{array} \right. \Rightarrow \left\{ \begin{array}{l}
A = \frac{1}{{\sqrt {4\alpha + 1} }} \\
B = - \frac{1}{{\sqrt {4\alpha + 1} }} \\
\end{array} \right.
\end{equation}
Substituting the result of formula (\ref{f15}) back into formula (\ref{f14}) yields $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\begin{equation}\label{f16}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2.5} % 控制行间距
\begin{array}{l}
\frac{1}{\alpha }\int {\frac{1}{{\sqrt {4\alpha + 1} \left( {u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}} \right)}}} - \frac{1}{{\sqrt {4\alpha + 1} \left( {u + \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}} \right)}} + \frac{1}{u}{\rm{d}}u \\
= \frac{1}{\alpha }\left( {\frac{1}{{\sqrt {4\alpha + 1} }}\underbrace {\int {\frac{1}{{u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}}}} {\rm{d}}u}_{(1)} - \frac{1}{{\sqrt {4\alpha + 1} }}\underbrace {\int {\frac{1}{{u - \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}}}} {\rm{d}}u}_{(2)} + \int {\frac{1}{u}{\rm{d}}u} } \right) \\
= \frac{{{\rm{ln}}\left( {\left| {2\alpha u + \sqrt {4\alpha + 1} + 2\alpha + 1} \right|} \right)}}{{\alpha \sqrt {4\alpha + 1} }} - \frac{{{\rm{ln}}\left( {\left| {2\alpha u - \sqrt {4\alpha + 1} + 2\alpha + 1} \right|} \right)}}{{\alpha \sqrt {4\alpha + 1} }} + \frac{{\ln \left( {\left| u \right|} \right)}}{\alpha } \\
= - \frac{{\ln \left( {|2\alpha {\rm{e}}^{\rm{x}} - \sqrt {4\alpha + 1} + 2\alpha + 1|} \right)}}{{\alpha \sqrt {4\alpha + 1} }} + \frac{{\ln \left( {|2\alpha {\rm{e}}^{\rm{x}} + \sqrt {4\alpha + 1} + 2\alpha + 1|} \right)}}{{\alpha \sqrt {4\alpha + 1} }} + \frac{x}{\alpha } + C \\
\end{array}
\end{equation}
For the (1) part in formula (\ref{f16}), we will solve it using the method of substitution. Let's assume $v = 2\alpha u + \sqrt {4\alpha + 1} + 2\alpha + 1$ as the substitution variable. Then $u = \frac{{v - \sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}$ as ${\rm{d}}u = \frac{1}{{2\alpha }}{\rm{d}}v$. Further steps for the computation are as follow
\[
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
\int {\frac{1}{{u + \frac{{\sqrt {4\alpha + 1} + 2\alpha + 1}}{{2\alpha }}}}} {\rm{d}}u \\
= \int {\frac{{2\alpha }}{{2\alpha u + \sqrt {4\alpha + 1} + 2\alpha + 1}}} {\rm{d}}u \\
= 2\alpha \int {\frac{1}{{2\alpha v}}} {\rm{d}}v \\
= \ln \left( {\left| v \right|} \right) \\
= \ln \left( {\left| {2\alpha u + \sqrt {4\alpha + 1} + 2\alpha + 1} \right|} \right) \\
\end{array}
\]
For the (2) part in formula (\ref{f16}), we will solve it using the method of substitution. Let's assume $v = 2\alpha u - \sqrt {4\alpha + 1} + 2\alpha + 1$ as the substitution variable. Then $u = \frac{{v + \sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}$. Further, we have ${\rm{d}}u = \frac{1}{{2\alpha }}{\rm{d}}v$. The specific computation process is as follows
\[
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\begin{array}{l}
\int {\frac{1}{{u - \frac{{\sqrt {4\alpha + 1} - 2\alpha - 1}}{{2\alpha }}}}} {\rm{d}}u \\
= \int {\frac{{2\alpha }}{{2\alpha u - \sqrt {4\alpha + 1} + 2\alpha + 1}}} {\rm{d}}u \\
= 2\alpha \int {\frac{1}{{2\alpha v}}} {\rm{d}}v \\
= \ln \left( {\left| v \right|} \right) \\
= \ln \left( {\left| {2\alpha u - \sqrt {4\alpha + 1} + 2\alpha + 1} \right|} \right) \\
\end{array}
\]
\subsection{Adversarial Functions of Other Activation Functions}
\subsubsection{Tanh Trainable Parameter Initialization} \label{A-2-1}
The partial derivative of $\xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$ can be obtained from formula (\ref{f11}), and the result of integrating each partial derivative is as follows
\begin{equation}\label{f17}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2.5} % 控制行间距
\left\{ \begin{array}{l}
\int {\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _1 }}} {\rm{d}}\chi _1 = {\rm{ - }}\left( {{\rm{e}}^{\chi _3 } + {\rm{e}}^{ - \chi _4 } } \right){\rm{e}}^{{\rm{ - }}\chi _{\rm{1}} } + C \\ \hspace{3.6cm} = - \left( {\alpha _1 + \alpha _2 } \right){\rm{e}}^{{\rm{ - }}\chi _{\rm{1}} } + \alpha _3 \\
\int {\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _2 }}} {\rm{d}}\chi _2 = {\rm{ - }}\left( {{\rm{e}}^{\chi _3 } + {\rm{e}}^{ - \chi _4 } } \right) \cdot {\rm{e}}^{{\rm{ - }}\chi _{\rm{2}} } + C \\ \hspace{3.6cm} = - \left( {\alpha _4 + \alpha _5 } \right){\rm{e}}^{{\rm{ - }}\chi _{\rm{1}} } + \alpha _6 \\
\int {\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _3 }}} {\rm{d}}\chi _3 = \frac{{{\rm{e}}^{\chi _2 } \left( {{\rm{e}}^{ - \chi _3 - 2\chi _4 } - {\rm{e}}^{\chi _3 } - 2{\rm{e}}^{ - \chi _4 } \chi _3 } \right)}}{{{\rm{e}}^{\chi _1 + \chi _2 } - 1}} + C \\ \hspace{3.6cm} = \frac{{\alpha _7 \left( {{\rm{e}}^{ - \chi _3 } \alpha _8 ^{ - 2} - {\rm{e}}^{\chi _3 } - 2\alpha _8 ^{ - 1} \chi _3 } \right)}}{{\alpha _7 \alpha _8 - 1}} + \alpha _9 \\
\int {\frac{{\partial \xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _4 }}} {\rm{d}}\chi _4 = \frac{{{\rm{e}}^{\chi _2 } \left( {{\rm{e}}^{\chi _4 + 2\chi _3 } - {\rm{e}}^{ - \chi _4 } - 2{\rm{e}}^{\chi _3 } \chi _4 } \right)}}{{{\rm{e}}^{\chi _1 + \chi _2 } - 1}} + C\\ \hspace{3.6cm} = \frac{{\alpha _{10} \left( {{\rm{e}}^{\chi _4 } \alpha _2 ^{\rm{2}} - {\rm{e}}^{\chi _4 } + 2\alpha _{12} \chi _4 } \right)}}{{\alpha _{10} \alpha _{11} - 1}} + \alpha _{12} \\
\end{array} \right.
\end{equation}
$\alpha_i$ in formula (\ref{f17}) represents trainable parameters. The trainable parameter in formula (\ref{f17}) can be initialized to the 1 unit constant calculated by the formula
\[
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{1.5} % 控制行间距
\left\{ \begin{array}{l}
\alpha _1 = \alpha _4 = {\rm{e}}^{\chi _3 } \Rightarrow {\rm{e}} \\
\alpha _2 = \alpha _5 = {\rm{e}}^{ - \chi _4 } \Rightarrow {\rm{e}} \\
\alpha _3 = \alpha _6 = \alpha _9 = \alpha _{12} = 0 \\
\alpha _7 = \alpha _{10} = {\rm{e}}^{\chi _2 } \Rightarrow {\rm{e}} \\
\alpha _8 = \alpha _{11} = e \\
\end{array} \right.
\]
\subsubsection{ The Adversarial Function of GeLu} \label{A-2-2}
\begin{figure}[h]
\centering
\begin{minipage}{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{GeLu.pdf}
\caption{HD-FGD splits GeLu into 4 terms.}
\label{fig:gleu}
\end{minipage}\hfill
\begin{minipage}{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{GeLu_com.pdf}
\caption{Calculation structure diagram of ${\xi _{{\rm{GeLu}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}$.}
\label{fig:gelu-com}
\end{minipage}
\end{figure}
GeLu is most widely used in models based on transformer architecture, with the expression $0.5x(1 + \rm{Tanh} (\sqrt {\frac{2}{\pi }} (x + 0.44715x^3 )))$, which is represented as formula (\ref{f18}) after passing through a linear layer
\begin{equation}\label{f18}
{\rm{GeLu}}(W,x) = 0.5Wx(1 + {\rm{Tanh}}(\sqrt {\frac{2}{\pi }} Wx(1 + Wx \cdot 0.44715Wx)))
\end{equation}
In the above formula, $W$ represents the linear layer parameter matrix. As shown in Figure \ref{fig:gleu}, splitting GeLu into 4 terms can be represented as
\begin{equation}\label{f19}
{\rm{GeLu}}(W_1 ,W_2 ,W_3 ,W_4 ,x) = 0.5W_1 x(1 + {\rm{Tanh}}(\sqrt {\frac{2}{{\rm{\pi }}}} W_2 x(1 + W_3 x \cdot 0.44715W_4 x)))
\end{equation}
Let $\chi _i = W_i x$, then formula (\ref{f19}) can be modified to
\[
{\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 ) = 0.5\chi _1 (1 + {\rm{Tanh}}(\sqrt {\frac{2}{{\rm{\pi }}}} \chi _2 (1 + \chi _3 \cdot 0.44715\chi _4 )))
\]
According to the network adversarial method, the partial derivative of ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3, \chi _4 )
$ can be solved as formula (\ref{f20})
\begin{equation}\label{f20}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2.5} % 控制行间距
\left\{ \begin{array}{l}
\frac{{\partial {\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _1 }} = 0.5\left( {{\rm{Tanh}}\left( {\sqrt {\frac{2}{\pi }\chi _2 } \left( {1 + \chi _3 \cdot 0.044715\chi _4 } \right)} \right)} \right) \\
\frac{{\partial {\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _2 }} = \frac{{8943\sqrt 2 \chi _1 \chi _3 \chi _4 + 3125.64\sqrt 2 \chi _1 }}{{400000\sqrt \pi \cosh ^2 \left( {\frac{{8943\chi _3 \chi _4 \chi _2 }}{{3125.32\sqrt {2\pi } }} + \sqrt {\frac{2}{\pi }} \chi _2 } \right)}} \\
\frac{{\partial {\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _3 }} = \frac{{8943\chi _1 \chi _2 \chi _4 }}{{3125.64\sqrt {2\pi } \cosh ^2 \left( {\frac{{8943\chi _2 \chi _4 \chi _3 }}{{3125.32\sqrt {2\pi } }} + \sqrt {\frac{2}{\pi }} \chi _2 } \right)}} \\
\frac{{\partial {\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\partial \chi _4 }} = \frac{{8943\chi _1 \chi _2 \chi _3 }}{{3125.64\sqrt {2\pi } \cosh ^2 \left( {\frac{{8943\chi _2 \chi _3 \chi _4 }}{{3125.32\sqrt {2\pi } }} + \sqrt {\frac{2}{\pi }} \chi _2 } \right)}} \\
\end{array} \right.
\end{equation}
Based on this, the partial derivative of $\xi _{{\rm{GeLu}}} \left( {\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 } \right)$ can be solved as the reciprocal of the partial derivatives of ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$ in formula (\ref{f20}), and the partial derivatives of $\xi _{{\rm{GeLu}}} \left( {\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 } \right)$ can be integrated separately
\begin{equation}\label{f21}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2.5} % 控制行间距
\left\{ \begin{array}{l}
\int {\frac{{\partial \xi _{{\rm{GeLu}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\chi _1 }}} {\rm{d}}\chi _1 = \frac{{\chi _1 }}{{0.5\left( {{\rm{1}} + {\rm{Tanh}}\left( {\sqrt {\frac{2}{\pi }} \chi _2 \left( {1 + \chi _3 \cdot 0.044715\chi _4 } \right)} \right)} \right)}} + C = \alpha _1 \chi _1 + \alpha _2 \\
\int {\frac{{\partial \xi _{{\rm{GeLu}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\chi _2 }}} {\rm{d}}\chi _2 = \frac{{17757500000\pi \cdot {\rm{sinh}}\left( {\frac{{{\rm{20325}}\sqrt {\rm{2}} \chi _3 \chi _4 \chi _2 }}{{7103\sqrt \pi }} + 2\sqrt {\frac{2}{\pi }} \chi _2 } \right)}}{{4544161875\chi _1 \chi _3 ^2 \chi _4 ^2 + 4764322275\chi _1 \chi _3 \chi _4 + 1110071041\chi _1 }} + \\
\hspace{4cm} \frac{{78125.32\sqrt {{\rm{2}}\pi } \chi _2 }}{{223575\chi _1 \chi _3 \chi _4 + 78141\chi _1 }} + C \\ \hspace{3.6cm}= \alpha _3 \sinh \left( {\alpha _4 \beta _1 \beta _2 \chi _2 + \alpha _5 \chi _2 } \right) + \alpha _6 \chi _2 + \alpha _7 \\
\int {\frac{{\partial \xi _{{\rm{GeLu}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\chi _3 }}} {\rm{d}}\chi _3 = \frac{{185011841\pi \cdot {\rm{sinh}}\left( {\frac{{{\rm{20325}}\sqrt {\rm{2}} \chi _2 \chi _4 \chi _3 }}{{7103\sqrt \pi }} + 2\sqrt {\frac{2}{\pi }} \chi _2 } \right)}}{{3029441250\chi _1 \chi _2 ^2 \chi _4 ^2 }} + \\ \hspace{4cm} \frac{{26047\sqrt \pi \chi _3 }}{{74525\sqrt {\rm{2}} \chi _1 \chi _2 \chi _4 }} + C \\
\hspace{3.6cm} = \alpha _8 \sinh \left( {\alpha _4 \beta _3 \beta _2 \chi _2 + \alpha _5 \chi _2 } \right) + \alpha _9 \chi _3 + \alpha _{10} \\
\int {\frac{{\partial \xi _{{\rm{GeLu}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}}{{\chi _4 }}} {\rm{d}}\chi _4 = \frac{{185011841\pi \cdot {\rm{sinh}}\left( {\frac{{{\rm{20325}}\sqrt {\rm{2}} \chi _2 \chi _3 \chi _4 }}{{7103\sqrt \pi }} + 2\sqrt {\frac{2}{\pi }} \chi _2 } \right)}}{{3029441250\chi _1 \chi _2 ^2 \chi _3 ^2 }} + \\ \hspace{4cm} \frac{{26047\sqrt \pi \chi _4 }}{{74525\sqrt {\rm{2}} \chi _1 \chi _2 \chi _3 }} + C \\
\hspace{3.6cm}= \alpha _{11} \sinh \left( {\alpha _4 \beta _3 \beta _1 \chi _2 + \alpha _5 \chi _2 } \right) + \alpha _{12} \chi _4 + \alpha _{13} \\
\end{array} \right.
\end{equation}
$\alpha_i$ and $\beta_i$ in formula (\ref{f21}) both represent trainable parameters.
% According to \ref{4-1}, in the case of GA, better results can be achieved if the zero-mean condition is satisfied. Therefore, in formula (\ref{f21}), the constant term represented by $C_\delta $ is set to 0 without adding trainable parameters to it. Additionally,
$\alpha_2$, $\alpha_7$, $\alpha_{10}$ and $\alpha_{13}$ are the constant terms obtained after integration, and they are set as trainable parameters initialized to 0. Other trainable parameters can be initialized to the constant of 1 unit calculated from the equation, as shown in formula (\ref{22})
\begin{equation}\label{22}
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\left\{ \begin{array}{l}
\alpha _1 = \frac{1}{{0.5\left( {{\rm{Tanh}}\left( {\sqrt {\frac{2}{\pi }} \chi _2 \left( {1 + \chi _3 \cdot 0.044715\chi _4 } \right)} \right)} \right)}} \Rightarrow \frac{1}{{0.5\left( {{\rm{Tanh}}\left( {\sqrt {\frac{2}{\pi }} \left( {1 + 0.044715} \right)} \right)} \right)}} \\
\alpha _3 = \frac{{17757500000\pi }}{{4544161875\chi _1 \chi _3 ^2 \chi _4 ^2 + 4764322275\chi _1 \chi _3 \chi _4 + 1110071041\chi _1 }} \Rightarrow \frac{{17757500000\pi }}{{10418555191}} \\
\alpha _4 = \frac{{20325\sqrt 2 }}{{7103\sqrt \pi }} \\
\beta _1 = \chi _3 \Rightarrow 1 \\
\beta _2 = \chi _4 \Rightarrow 1 \\
\beta _3 = \chi _2 \Rightarrow 1 \\
\alpha _5 = 2\sqrt {\frac{2}{\pi }} \\
\alpha _6 = \frac{{78125.32\sqrt {2\pi } \chi _2 }}{{223575\chi _1 \chi _3 \chi _4 + 78141\chi _1 }} \Rightarrow \frac{{78125.32\sqrt {2\pi } }}{{301716}} \\
\alpha _8 = \frac{{185011841\pi }}{{3029441250\chi _1 \chi _2 ^2 \chi _4 ^2 }} \Rightarrow \frac{{185011841\pi }}{{3029441250}} \\
\alpha _9 = \frac{{26047\sqrt \pi \chi _2 }}{{74525\sqrt 2 \chi _1 \chi _2 \chi _4 }} \Rightarrow \frac{{26047\sqrt \pi }}{{74525\sqrt 2 }} \\
\alpha _{11} = \frac{{185011841\pi }}{{3029441250\chi _1 \chi _2 ^2 \chi _3 ^2 }} \Rightarrow \frac{{185011841\pi }}{{3029441250}} \\
\alpha _{12} = \frac{{26047\sqrt \pi \chi _4 }}{{74525\sqrt 2 \chi _1 \chi _2 \chi _3 }} \Rightarrow \frac{{26047\sqrt \pi }}{{74525\sqrt 2 }} \\
\end{array} \right.
\end{equation}
$\xi _{{\rm{GeLu}}} (\chi _1 ,\chi _2 ,\chi _3, \chi _4 )$ can be represented as
\[
\begin{array}{l}
\xi _{{\rm{GeLu}}} \left( {\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 } \right) =\left({\alpha _1 \chi _1 + \alpha _2 } \right) \cdot \left( {\alpha _3 {\rm{sinh}}\left( {\alpha _4 \beta _1 \beta _2 \chi _2 + \alpha _5 \chi _2 } \right) + \alpha _6 \chi _2 + \alpha _7 } \right) \cdot \\
\begin{array}{*{20}c}
{\begin{array}{*{20}c}
{\begin{array}{*{20}c}
{} & {} \\
\end{array}} & {} & {} & {} \\
\end{array}} & {} & {} & {} \hspace{0.5mm} \\
\end{array}\left( {\alpha _8 {\rm{sinh}}\left( {\alpha _4 \beta _2 \beta _3 \chi _3 + \alpha _5 \chi _2 } \right) + \alpha _9 \chi _3 + \alpha _{10} } \right) \cdot \\
\begin{array}{*{20}c}
{\begin{array}{*{20}c}
{\begin{array}{*{20}c}
{} & {} \\
\end{array}} & {} & {} & {} \\
\end{array}} & {} & {} & {} \hspace{0.5mm} \\
\end{array}\left( {\alpha _{11} {\rm{sinh}}\left( {\alpha _4 \beta _1 \beta _3 \chi _4 + \alpha _5 \chi _2 } \right) + \alpha _{12} \chi _4 + \alpha _{13} } \right) \\
\end{array}
\]
In the splitting of formula (\ref{f21}), considering the complexity of the function and the possibility of overfitting with too many learnable parameters. As shown in Figure \ref{fig:gelu-com}, we tend to write the general formula of the function, simplify some learnable parameters, and reinitialize them as
\[
\setlength{\arraycolsep}{2pt} % 控制列间距
\renewcommand{\arraystretch}{2} % 控制行间距
\left\{ \begin{array}{l}
\delta _1 = \frac{{\left( {\alpha _3 + \alpha _8 + \alpha _{11} } \right)}}{3} \\
\delta _2 = \frac{{\left( {\alpha _6 + \alpha _9 + \alpha _{12} } \right)}}{3} \\
\delta _3 = \frac{{\left( {\alpha _7 + \alpha _{10} + \alpha _{13} } \right)}}{3} \\
\beta = \beta _1 \beta _2 \beta _3\\
\chi = \frac{{\left( {\chi _2 + \chi _3 + \chi _{4} }
\right)}}{3}\\
\end{array} \right.
\]
$\xi _{{\rm{GeLu}}} \left({\chi _1 ,\chi _2 ,\chi _3, \chi _4} \right)$ can be rephrased as $\xi _{{\rm{GeLu}}} (\chi _1 ,\chi)$.
\[
\xi _{{\rm{GeLu}}} \left( {\chi _1 ,\chi} \right) = \left( {\alpha _1 \chi _1 + \alpha _2 } \right) \cdot \left( {\delta _1 {\rm{sinh}}\left( {\alpha _4 \beta \chi + \alpha _5 \beta _2 } \right) + \delta _2 \chi + \delta _3 } \right)
\]
\subsection{Analysis of Experimental Results}
\subsubsection{Analysis of GA Experiment Results} \label{A-3-1}
On the CIFAR10 dataset, using Sigmoid and its overall adversarial counterpart as the activation functions for the network layers of the VIT, SwT, and ResNet models resulted in improved prediction performance to varying degrees. Among them, ResNet showed the most significant improvement, with an increase of 101.52\% in ACC and a decrease of 82.00\% in Top-5 Error. In the CIFAR100 dataset, when Sigmoid was used as the activation function for the ResNet model in the classification task, it was almost ineffective in ResNet34. However, after applying the GA based on Sigmoid, there was a remarkable improvement in ACC with an increase of 1466.11\%, and a considerable decrease in Top-5 Error by 73.97\%.
\begin{table}
\centering
\caption{ACC and Top-5 Error based on Sigmoid GA. GA represents global adversarial.}
\begin{tabular}{ccccc}
\toprule
\textbf{Dataset} & \textbf{Model} & \textbf{GA} & \textbf{ACC} & \textbf{Top-5 Error} \\
\midrule
\multirow{6}{*}{\raisebox{-1.5ex}[0pt][0pt]{\rotatebox[origin=c]{90}{\textbf{CIFAR10}}}} & \multirow{2}{*}{VIT} & \checkmark & \textbf{63.26 $\pm$ 0.15} & \textbf{3.12 $\pm$ 0.12} \\
& & $\times$ & 55.17 $\pm$ 0.00 & 4.84 $\pm$ 0.00 \\
\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}
& \multirow{2}{*}{ResNet} & \checkmark & \textbf{75.53 $\pm$ 0.32} & \textbf{1.55 $\pm$ 0.73} \\
& & $\times$ & 37.48 $\pm$ 2.21 & 8.61 $\pm$ 0.07 \\
\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}
& \multirow{2}{*}{SwT} & \checkmark & \textbf{50.14 $\pm$ 0.29} & \textbf{6.91 $\pm$ 0.23} \\
& & $\times$ & 48.27 $\pm$ 1.09 & 7.43 $\pm$ 0.45 \\
\midrule
\multirow{6}{*}{\raisebox{-1.5ex}[0pt][0pt]{\rotatebox[origin=c]{90}{\textbf{CIFAR100}}}} & \multirow{2}{*}{VIT} & \checkmark & \textbf{39.41 $\pm$ 0.00} & \textbf{29.86 $\pm$ 0.00} \\
& & $\times$ & 29.66 $\pm$ 0.00 & 39.95 $\pm$ 0.00 \\
\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}
& \multirow{2}{*}{ResNet} & \checkmark & \textbf{47.14 $\pm$ 0.56} & \textbf{22.76 $\pm$ 0.32} \\
& & $\times$ & 3.01 $\pm$ 0.18 & 87.44 $\pm$ 0.65 \\
\cmidrule(lr){2-2}\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}
& \multirow{2}{*}{SwT} & \checkmark & \textbf{24.41 $\pm$ 0.45} & \textbf{47.76 $\pm$ 0.40} \\
& & $\times$ & 19.56 $\pm$ 0.29 & 53.71 $\pm$ 0.45 \\
\bottomrule
\end{tabular}
\label{table-1}
\end{table}
\begin{figure}[h]
\centering
\begin{minipage}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{ResNet-Sigmoid.png}
\subcaption{ResNet + Sigmoid}
% \label{fig:4-a}
\end{minipage}
\hfill
\begin{minipage}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{Swin_transformer_tiny-Sigmoid.png}
\subcaption{SwT + Sigmoid}
% \label{fig:4-b}
\end{minipage}
\hfill
\begin{minipage}[b]{0.32\textwidth}
\centering
\includegraphics[width=\textwidth]{VIT-Sigmoid.png}
\subcaption{VIT + Sigmoid}
% \label{fig:4-c}
\end{minipage}
\caption{Compared with Sigmoid, the GA test losses of Sigmoid in different scenario tasks.}
\label{fig:4}
\end{figure}
\subsubsection{Analysis of HD-FGD Experiment Results} \label{A-3-2}
In this experiment, Tanh and GeLu were used as activation function in the VIT and SwT models, respectively. The models were tested on the CIFAR10 and CIFAR100 datasets, and the results were used as the baseline. The HD-FGD was applied to split Tanh and GeLu into four components, and the decomposed forms were used to replace the original activation function.
The results in Table \ref{table-2} showed that, for the CIFAR10 classification task, using HD-FGD resulted in ACC improvements ranging from 1.04\% to 68.89\%, Top-5 Error reductions ranging from 0.32\% to 73.22\%, and average runtime reductions per epoch ranging from 0.17\% to 10.41\%. For the CIFAR100 classification task, ACC improvements ranged from 3.77\% to 37.47\%, Top-5 Error reductions ranged from 2.10\% to 27.47\%, and average runtime reductions per epoch ranged from 2.17\% to 9.15\%.
Figure \ref{fig:5} displayed the loss curves on the test set for SwT and VIT models with Tanh as the activation function and the decomposed form of Tanh obtained through HD-FGD. Both curves using HD-FGD were noticeably lower than the original activation function. Similarly, the loss convergence curves for GeLu after applying HD-FGD were lower than the baseline in all models, as shown in Figure \ref{fig:6}.
\begin{table}
\centering
\caption{ACC, Top-5 Error, and Epoch Time based on HD-FGD. AF represents activation function, HD-FGD represents high-dimensional function graph decomposition.}
\begin{tabular}{ccccccc}
\toprule
\textbf{Dataset} & \textbf{Model} & \textbf{AF} & \textbf{HD-FGD} & \textbf{ACC} & \textbf{Top-5 Error} & \textbf{Epoch Time} \\
\midrule
\multirow{8}{*}{\raisebox{-1.5ex}[0pt][0pt]{\rotatebox[origin=c]{90}{\textbf{CIFAR10}}}} & \multirow{4}{*}{VIT} & \multirow{2}{*}{Tanh} & $\checkmark$ & \textbf{75.23$\pm$0.27} & \textbf{1.49$\pm$0.07} & \textbf{89.38$\pm$0.93} \\
& & & $\times$ & 64.64$\pm$0.28 & 2.83$\pm$0.24 & 104.55$\pm$0.14 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}
& & \multirow{2}{*}{GeLu} & $\checkmark$ & \textbf{72.22$\pm$0.69} & \textbf{1.76$\pm$0.08} & \textbf{88.11$\pm$0.18} \\
& & & $\times$ & 64.53$\pm$0.43 & 3.27$\pm$0.19 & 140.90$\pm$0.24 \\
\cmidrule(r){2-7}
& \multirow{4}{*}{SwT} & \multirow{2}{*}{Tanh} & $\checkmark$ & \textbf{79.00$\pm$0.28} & \textbf{1.03$\pm$0.07} & \textbf{20.07$\pm$0.03} \\
& & & $\times$ & 60.91$\pm$0.35 & 3.86$\pm$0.11 & 22.87$\pm$2.63 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}
& & \multirow{2}{*}{GeLu} & $\checkmark$ & \textbf{83.74$\pm$0.21} & \textbf{0.64$\pm$0.02} & \textbf{20.58$\pm$0.04} \\
& & & $\times$ & 82.77$\pm$0.25 & 0.71$\pm$0.07 & 26.60$\pm$4.09 \\
\midrule
\multirow{8}{*}{\raisebox{-1.5ex}[0pt][0pt]{\rotatebox[origin=c]{90}{\textbf{CIFAR100}}}} & \multirow{4}{*}{VIT} & \multirow{2}{*}{Tanh} & $\checkmark$ & \textbf{48.62$\pm$0.16} & \textbf{23.29$\pm$0.24} & \textbf{91.00$\pm$0.19} \\
& & & $\times$ & 41.36$\pm$0.44 & 27.47$\pm$0.35 & 103.14$\pm$1.41 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}
& & \multirow{2}{*}{GeLu} & $\checkmark$ & \textbf{50.88$\pm$0.54} & \textbf{20.43$\pm$0.41} & \textbf{89.85$\pm$0.10} \\
& & & $\times$ & 47.92$\pm$0.29 & 23.51$\pm$0.18 & 102.05$\pm$0.13 \\
\cmidrule(r){2-7}
& \multirow{4}{*}{SwT} & \multirow{2}{*}{Tanh} & $\checkmark$ & \textbf{49.45$\pm$0.22} & \textbf{20.90$\pm$0.24} & \textbf{20.41$\pm$0.01} \\
& & & $\times$ & 29.28$\pm$0.42 & 41.54$\pm$0.29 & 21.21$\pm$0.31 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}
& & \multirow{2}{*}{GeLu} & $\checkmark$ & \textbf{57.31$\pm$0.40} & \textbf{15.65$\pm$0.18} & \textbf{20.92$\pm$0.03} \\
& & & $\times$ & 56.72$\pm$0.25 & 15.70$\pm$0.21 & 24.39$\pm$0.09 \\
\bottomrule
\end{tabular}
\label{table-2}
\end{table}
\begin{figure}[h]
\centering
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Swin_transformer_tiny-Tanh-HD-FGD.png}
\subcaption{SwT + Tanh}
% \label{fig:5-a}
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{VIT_Base-Tanh-HD-FGD.png}
\subcaption{VIT + Tanh}
% \label{fig:5-b}
\end{minipage}
\caption{Compared with Tanh, HD-FGD test losses of Tanh in different scenario tasks.}
\label{fig:5}
\end{figure}
\begin{figure}[h]
\centering
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Swin_transformer_tiny-GELU-HD-FGD.png}
\subcaption{SwT + GeLu}
% \label{fig:6-a}
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{VIT_Base-GELU-HD-FGD.png}
\subcaption{VIT + GeLu}
% \label{fig:6-b}
\end{minipage}
\caption{Compared with GeLu, HD-FGD test losses of GeLu in different scenario tasks.}
\label{fig:6}
\end{figure}
\subsubsection{Analysis of SA Experiment Results} \label{A-3-3}
In addition to using HD-FGD, further adversarial decomposition was applied by obtaining adversarial functions for Tanh and GeLu through network adversarial method. These adversarial functions, along with the decomposed activation functions obtained through HD-FGD, were alternately applied to the network layers of the models.
The results in Table \ref{table-3} indicate that using HD-FGD or SA to replace MLP and combine it with the original activation function will bring greater gains to the model. In addition to the improvement in ACC and Top5 Error, the training time for each epoch of the model is also significantly reduced. Generally, HD-FGD will bring various performance improvements to the model, while the network adversarial approach brings greater gains in ACC and Top-5 Error. Although the training time may increase slightly, it still significantly shortens the model training time compared to the experimental benchmark.
On the CIFAR10 dataset, adversarial decomposition led to ACC improvements ranging from 3.09\% to 40.72\% and Top-5 Error reductions ranging from 22.54\% to 86.01\% , and average runtime reductions per epoch ranging from 3.63\% to 29.39\% for the various models. On the CIFAR100 dataset, adversarial decomposition resulted in ACC improvements ranging from 5.45\% to 73.91\% and Top-5 Error reductions ranging from 4.76\% to 65.62\% , and average runtime reductions per epoch ranging from 1.87\% to 15.17\% for the different models.
Figure \ref{fig:7} and Figure \ref{fig:8} depicted the loss curves on the test set for the SwT and VIT models when using the adversarial decomposition based on Tanh and GeLu. In all experiments, the loss curve of the model using SA was significantly more stable and lower, indicating better predictive performance and faster convergence speed. It is important to note that Tanh and GeLu adopted the same decomposition form in this experiment. Exploring other decompositions based on different scenarios could potentially lead to even better performance when the task complexity increases.
\begin{table}[h]
\centering
\caption{ACC, Top-5 Error and Epoch Time based on SA. AF represents activation function, HD-FGD represents high-dimensional function graph decomposition and SA represents split adversarial.}
\begin{tabular}{cccccccc}
\toprule
\textbf{Dataset} & \textbf{Model} & \textbf{AF} & \textbf{HD-FGD} & \textbf{SA} & \textbf{ACC} & \textbf{Top-5 Error} & \textbf{Epoch Time} \\
\midrule
\multirow{12}{*}{\rotatebox[origin=c]{90}{\textbf{CIFAR10}}} & \multirow{6}{*}{VIT} & \multirow{3}{*}{Tanh} & $\checkmark$ & $\checkmark$ & \textbf{78.35}$\pm$\textbf{0.43} & \textbf{1.13}$\pm$\textbf{0.07} & 94.17$\pm$0.08 \\
& & & $\checkmark$ & $\times$ & 75.23$\pm$0.27 & 1.49$\pm$0.07 & \textbf{89.38}$\pm$\textbf{0.93} \\
& & & $\times$ & $\times$ & 64.64$\pm$0.28 & 2.83$\pm$0.24 & 104.54$\pm$0.14 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}\cmidrule(lr){8-8}
& & \multirow{3}{*}{GeLu} & $\checkmark$ & $\checkmark$ & \textbf{78.77}$\pm$\textbf{0.33} & \textbf{1.11}$\pm$\textbf{0.07} & 99.49$\pm$0.18 \\
& & & $\checkmark$ & $\times$ & 72.22$\pm$0.69 & 1.76$\pm$0.08 & \textbf{88.11}$\pm$\textbf{0.18} \\
& & & $\times$ & $\times$ & 64.53$\pm$0.43 & 3.27$\pm$0.19 & 140.90$\pm$0.24 \\
\cmidrule(lr){2-8}
& \multirow{6}{*}{SwT} & \multirow{3}{*}{Tanh} & $\checkmark$ & $\checkmark$ & \textbf{85.71}$\pm$\textbf{0.26} & \textbf{0.54}$\pm$\textbf{0.05} & 22.04$\pm$0.03 \\
& & & $\checkmark$ & $\times$ & 79.00$\pm$0.28 & 1.03$\pm$0.07 & \textbf{20.07}$\pm$\textbf{0.03} \\
& & & $\times$ & $\times$ & 60.91$\pm$0.35 & 3.86$\pm$0.11 & 22.87$\pm$2.63 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}\cmidrule(lr){8-8}
& & \multirow{3}{*}{GeLu} & $\checkmark$ & $\checkmark$ & \textbf{85.33}$\pm$\textbf{0.21} & \textbf{0.55}$\pm$\textbf{0.04} & \textbf{20.35}$\pm$\textbf{0.05} \\
& & & $\checkmark$ & $\times$ & 83.74$\pm$0.21 & 0.64$\pm$0.02 & 20.58$\pm$0.04 \\
& & & $\times$ & $\times$ & 82.77$\pm$0.25 & 0.71$\pm$0.07 & 26.60$\pm$4.09 \\
\midrule
\multirow{12}{*}{\rotatebox[origin=c]{90}{\textbf{CIFAR100}}} & \multirow{6}{*}{VIT} & \multirow{3}{*}{Tanh} & $\checkmark$ & $\checkmark$ & \textbf{49.79}$\pm$\textbf{1.01} & \textbf{22.22}$\pm$\textbf{0.76} & 96.22$\pm$0.04 \\
& & & $\checkmark$ & $\times$ & 48.62$\pm$0.16 & 23.29$\pm$0.24 & \textbf{91.00}$\pm$\textbf{0.19} \\
& & & $\times$ & $\times$ & 41.36$\pm$0.44 & 27.47$\pm$0.35 & 103.14$\pm$1.41 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}\cmidrule(lr){8-8}
& & \multirow{3}{*}{GeLu} & $\checkmark$ & $\checkmark$ & \textbf{50.92}$\pm$\textbf{0.41} & 22.39$\pm$0.33 & 100.14$\pm$1.02 \\
& & & $\checkmark$ & $\times$ & 50.88$\pm$0.54 & \textbf{20.43}$\pm$\textbf{0.41} & \textbf{89.85}$\pm$\textbf{0.10} \\
& & & $\times$ & $\times$ & 47.92$\pm$0.29 & 23.51$\pm$0.18 & 102.05$\pm$0.13 \\
\cmidrule(lr){2-8}
& \multirow{6}{*}{SwT} & \multirow{3}{*}{Tanh} & $\checkmark$ & $\checkmark$ & \textbf{59.92}$\pm$\textbf{0.13} & \textbf{14.28}$\pm$\textbf{0.11} & 22.35$\pm$0.03 \\
& & & $\checkmark$ & $\times$ & 49.45$\pm$0.22 & 20.9$\pm$0.24 & \textbf{20.41}$\pm$\textbf{0.01} \\
& & & $\times$ & $\times$ & 29.28$\pm$0.42 & 41.54$\pm$0.29 & 21.21$\pm$0.31 \\
\cmidrule(lr){3-3}\cmidrule(lr){4-4}\cmidrule(lr){5-5}\cmidrule(lr){6-6}\cmidrule(lr){7-7}\cmidrule(lr){8-8}
& & \multirow{3}{*}{GeLu} & $\checkmark$ & $\checkmark$ & \textbf{59.81}$\pm$\textbf{0.14} & \textbf{14.91}$\pm$\textbf{0.20} & \textbf{20.69}$\pm$\textbf{0.02} \\
& & & $\checkmark$ & $\times$ & 57.31$\pm$0.40 & 15.65$\pm$0.18 & 20.92$\pm$0.03 \\
& & & $\times$ & $\times$ & 56.72$\pm$0.25 & 15.70$\pm$0.21 & 24.39$\pm$0.09 \\
\bottomrule
\end{tabular}
\label{table-3}
\end{table}
\begin{figure}[h]
\centering
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Swin_transformer_tiny-Tanh-SA.png}
\subcaption{SwT + Tanh}
% \label{fig:7-a}
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{VIT_Base-Tanh-SA.png}
\subcaption{VIT + Tanh}
% \label{fig:7-b}
\end{minipage}
\caption{Compared with Tanh, SA test losses of Tanh in different scenario tasks.}
\label{fig:7}
\end{figure}
\begin{figure}[h]
\centering
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Swin_transformer_tiny-GeLu-SA.png}
\subcaption{SwT + GeLu}
% \label{fig:8-a}
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{VIT_Base-GeLu-SA.png}
\subcaption{VIT + GeLu}
% \label{fig:8-b}
\end{minipage}
\caption{Compared with GeLu, SA test losses of GeLu in different scenario tasks.}
\label{fig:8}
\end{figure}
\subsection{Experimental Details}
In this section, we will provide the network architecture and hyperparameters used for experiments in \ref{5}. For all experiments, the following settings will be used.
\begin{itemize}
\item Epochs: 150
\item Batch Size: 128
\item Optimizer: SGD
\item Learning Rate: 0.1
\item Learning Rate Scheduler: $\left(0.5^{ \left(\frac{\text{epoch}}
{20}\right)} \right) \times $ Learning Rate
\item MLP Ratio: 4
\end{itemize}
In order to maintain the same number of parameters as the benchmark model using network adversarial method, it is necessary to strictly limit the setting of the activation function splitting parameter Number of Parallel Matrices. The specific calculation method is as follows.
Assuming that the number of splitting terms for the activation function is $k$ and the Number of Parallel Matrices is $n$, then there is
\begin{equation}\label{f23}
\begin{array}{l}
\psi _{\rm{I}} \times \psi _{\rm{H}} + \psi _{\rm{H}} \times \psi _{\rm{O}} = \psi _{\rm{I}} \times \frac{{\psi _{\rm{H}} }}{n} \times k + \frac{{\psi _{\rm{H}} }}{n} \times \psi _{\rm{O}}
\Rightarrow n = \frac{{\psi _{\rm{I}} \times \psi _{\rm{H}} \times k + \psi _{\rm{H}} \times \psi _{\rm{O}} }}{{\psi _{\rm{I}} \times \psi _{\rm{H}} + \psi _{\rm{H}} \times \psi _{\rm{O}} }} \\
\end{array}
\end{equation}
Among them, $\psi _{\rm{I}} $ represents the input feature dimension of the multi-layer perceptron, $\psi _{\rm{H}} $ represents the hidden layer feature dimension of the multi-layer perceptron, and $\psi _{\rm{O}} $ represents the output feature dimension of the multi-layer perceptron. In the VIT and SwT experiments involved in this research
\begin{equation}\label{f24}
\left\{ \begin{array}{l}
\psi _{\rm{I}} = {\rm{Embedding ~ Dimension}} \\
\psi _{\rm{H}} = {\rm{Embedding ~ Dimension}} \times {\rm{MLP ~ Ratio}} \\
\psi _{\rm{O}} = {\rm{Embedding ~ Dimension}} \\
\end{array} \right.
\end{equation}
Among them, Embedding Dimension represents the output dimension of the embedding layer, and MLP Ratio represents the ratio of the width of the hidden layer to its input width in a multi-layer perceptron. By substituting formula (\ref{f24}) into formula (\ref{f23}), the solution is $n = \frac{{k + 1}}{2}$. Therefore, when the original activation function is split into 3 terms, $n=2$, and when the activation function is split into 4 terms,$ n=2.5$. In the experiments of this paper, $n=4$, which results in fewer parameters compared to the baseline model.
Whether to add batch normalization in the network layer is based on the decision criterion of having the best effect on the normal activation function model. For other details of different models under different scene tasks, the following settings are provided.
For \textbf{CIFAR10} dataset, common settings as follows.
Hyperparameters for \textbf{ResNet with Sigmoid} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Sigmoid(x)}$
\end{itemize}
Hyperparameters for \textbf{ResNet with Sigmoid GA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{Sigmoid}}_\vartheta (x)$ + $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\end{itemize}
For all SwT experiments, the following settings were used
\begin{itemize}
\item Patch Size: 4
\item Window Size: 7
\item Embedding Dimension:96
\item Depths: (2, 2, 6, 2)
\item Num Heads: (3, 6, 12, 24)
\end{itemize}
Hyperparameters for \textbf{SwT with Sigmoid} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Sigmoid(x)}$
\end{itemize}
Hyperparameters for \textbf{SwT with Sigmoid GA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{Sigmoid}}_\vartheta (x)$ + $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\end{itemize}
Hyperparameters for \textbf{SwT with GeLu} are as follows.
\begin{itemize}
\item Activation Function: $\rm{GeLu(x)}$
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with Tanh} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with Tanh after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(\chi _1 ,\chi _2 ,\chi _3, \chi _4 )}$
\item SA: False
\item Number of Parallel Matrices: 4
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with Tanh SA} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$ + $\xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 , \chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.05
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with GeLu after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3, \chi _4 )$
\item SA: False
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.005
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with GeLu SA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3, \chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.05
\item Batch Normalization: True
\end{itemize}
For all VIT experiments, the following settings were used
\begin{itemize}
\item Patch Size: 4
\item Embedding Dimension:768
\item Depth: 12
\item Num Heads: 12
\end{itemize}
Hyperparameters for \textbf{VIT with Sigmoid} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Sigmoid(x)}$
\end{itemize}
Hyperparameters for \textbf{VIT with Sigmoid GA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{Sigmoid}}_\vartheta (x)$ + $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\end{itemize}
Hyperparameters for \textbf{VIT with GeLu} are as follows.
\begin{itemize}
\item Activation Function: $\rm{GeLu(x)}$
\end{itemize}
Hyperparameters for \textbf{VIT with Tanh} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$
\end{itemize}
Hyperparameters for \textbf{VIT with Tanh after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )}$
\item SA: False
\item Number of Parallel Matrices: 4
\end{itemize}
Hyperparameters for \textbf{VIT with Tanh SA} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$ + $\xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3,\chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.05
\end{itemize}
Hyperparameters for \textbf{VIT with GeLu after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$
\item SA: False
\item Number of Parallel Matrices: 4
\end{itemize}
Hyperparameters for \textbf{VIT with GeLu SA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.05
\end{itemize}
For \textbf{CIFAR100} dataset, common settings as follows.
Hyperparameters for \textbf{ResNet with Sigmoid} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Sigmoid(x)}$
\end{itemize}
Hyperparameters for \textbf{ResNet with Sigmoid GA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{Sigmoid}}_\vartheta (x)$ + $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\end{itemize}
For all SwT experiments, the following settings were used
\begin{itemize}
\item Patch Size: 4
\item Window Size: 7
\item Embedding Dimension:96
\item Depths: (2, 2, 6, 2)
\item Num Heads: (3, 6, 12, 24)
\end{itemize}
Hyperparameters for \textbf{SwT with Sigmoid} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Sigmoid(x)}$
\end{itemize}
Hyperparameters for \textbf{SwT with Sigmoid GA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{Sigmoid}}_\vartheta (x)$ + $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\end{itemize}
Hyperparameters for \textbf{SwT with GeLu} are as follows.
\begin{itemize}
\item Activation Function: $\rm{GeLu(x)}$
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with Tanh} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with Tanh after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(\chi _1 ,\chi _2 ,\chi _3,\chi _4 )}$
\item SA: False
\item Number of Parallel Matrices: 4
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with Tanh SA} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$ + $\xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.05
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with GeLu after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 )$
\item SA: False
\item Number of Parallel Matrices: 3
\item L2 Penalty Coefficient: 0.005
\item Batch Normalization: True
\end{itemize}
Hyperparameters for \textbf{SwT with GeLu SA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3,\chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.05
\item Batch Normalization: True
\end{itemize}
For all VIT experiments, the following settings were used
\begin{itemize}
\item Patch Size: 4
\item Embedding Dimension:768
\item Depth: 12
\item Num Heads: 12
\end{itemize}
Hyperparameters for \textbf{VIT with Sigmoid} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Sigmoid(x)}$
\end{itemize}
Hyperparameters for \textbf{VIT with Sigmoid GA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{Sigmoid}}_\vartheta (x)$ + $\xi _{{\rm{Sigmoid}}_\vartheta } (x)$
\end{itemize}
Hyperparameters for \textbf{VIT with GeLu} are as follows.
\begin{itemize}
\item Activation Function: $\rm{GeLu(x)}$
\end{itemize}
Hyperparameters for \textbf{VIT with Tanh} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$
\end{itemize}
Hyperparameters for \textbf{VIT with Tanh after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(\chi _1 ,\chi _2 ,\chi _3, \chi _4 )}$
\item SA: False
\item Number of Parallel Matrices: 4
\end{itemize}
Hyperparameters for \textbf{VIT with Tanh SA} are as follows.
\begin{itemize}
\item Activation Function: $\rm{Tanh(x)}$ + $\xi _{{\rm{Tanh}}} (\chi _1 ,\chi _2 ,\chi _3 ,\chi _4 )$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.15
\end{itemize}
Hyperparameters for \textbf{VIT with GeLu after HD-FGD} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3,\chi _4 )$
\item SA: False
\item Number of Parallel Matrices: 4
\end{itemize}
Hyperparameters for \textbf{VIT with GeLu SA} are as follows.
\begin{itemize}
\item Activation Function: ${\rm{GeLu}}(\chi _1 ,\chi _2 ,\chi _3 ,\chi _4)$
\item SA: True
\item Number of Parallel Matrices: 4
\item L2 Penalty Coefficient: 0.08
\end{itemize}
\subsection{Adversarial Method in the Field of NLP Have Been Proven to be Effective} \label{A-4}
Translation of network adversarial method to test their effectiveness in the NLP field involves experiments using an LSTM-based autoencoder network with Sigmoid activation functions to create a baseline Seq2Seq model. In these experiments, the number of epochs is set to 50, the learning rate to 0.001, and the batch size to 128. Both the encoder and decoder have an embedding size of 300, and the output factor for both is 0.5. Training is conducted using the Adam optimizer for cross-entropy loss. We replicated some experiments from Table 11 in reference \cite{dubey2022activation}, with all experimental results averaging over five random seeds. Before the dropout layer, the activation function is applied to the feature embeddings. Experimental results in \ref{table-4} show that the Sigmoid activation function resulted in the poorest translation performance, while GeLu yielded the best translation results. By applying adversarial method to the network, using its adversarial function as the activation function in the LSTM's Encoder and Decoder, the final model's translation performance was better than that using GeLu as the activation function.
\begin{table}[h]
\centering
\caption{Experimental results for German to English language translation.}
\begin{tabular}{ccc}
\toprule
\multicolumn{2}{c}{\textbf{~Activation}} & \multirow{2}{*}{\textbf{Bleu Score}} \\
% \cline{1-2}
\cmidrule(lr){1-2}
\multicolumn{1}{l}{Encoder} & \multicolumn{1}{l}{Decoder} & \\
\hline
Sigmoid & Sigmoid & 14.57$\pm$0.83 \\
$\xi _{{\rm{Sigmoid}}} \left( x \right)$ & $\xi _{{\rm{Sigmoid}}} \left( x \right)$ & \textbf{21.88}$\pm$\textbf{0.35} \\
GeLu & GeLu & 21.30$\pm$0.29 \\
ReLu & ReLU & 20.81$\pm$0.34 \\
Tanh & Tanh & 21.10$\pm$0.40 \\
\bottomrule
\end{tabular}
\label{table-4}
\end{table}
This demonstrates the potential of network adversarial methods in the NLP field, and we look forward to exploring its effectiveness in more areas and believe in further advancements in the future.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% % \newpage
% \clearpage
% \section*{NeurIPS Paper Checklist}
% %%% BEGIN INSTRUCTIONS %%%
% The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page
% limit.
% Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
% \begin{itemize}
% \item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
% \item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
% \item Please provide a short (1–2 sentence) justification right after your answer (even for NA).
% % \item {\bf The papers not including the checklist will be desk rejected.}
% \end{itemize}
% {\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
% The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "\answerYes{}" is generally preferable to "\answerNo{}", it is perfectly acceptable to answer "\answerNo{}" provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "\answerNo{}" or "\answerNA{}" is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
% IMPORTANT, please:
% \begin{itemize}
% \item {\bf Delete this instruction block, but keep the section heading ``NeurIPS paper checklist"},
% \item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
% \item {\bf Do not modify the questions and only use the provided macros for your answers}.
% \end{itemize}
% %%% END INSTRUCTIONS %%%
% \begin{enumerate}
% \item {\bf Claims}
% \item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the abstract and introduction do not include the claims made in the paper.
% \item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
% \item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
% \item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
% \end{itemize}
% \item {\bf Limitations}
% \item[] Question: Does the paper discuss the limitations of the work performed by the authors?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
% \item The authors are encouraged to create a separate "Limitations" section in their paper.
% \item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
% \item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
% \item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
% \item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
% \item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
% \item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
% \end{itemize}
% \item {\bf Theory Assumptions and Proofs}
% \item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include theoretical results.
% \item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
% \item All assumptions should be clearly stated or referenced in the statement of any theorems.
% \item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
% \item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
% \item Theorems and Lemmas that the proof relies upon should be properly referenced.
% \end{itemize}
% \item {\bf Experimental Result Reproducibility}
% \item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
% \item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
% \item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
% \item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
% \begin{enumerate}
% \item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
% \item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
% \item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
% \item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
% \end{enumerate}
% \end{itemize}
% \item {\bf Open access to data and code}
% \item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that paper does not include experiments requiring code.
% \item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
% \item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
% \item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
% \item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
% \item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
% \end{itemize}
% \item {\bf Experimental Setting/Details}
% \item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
% \item The full details can be provided either with the code, in appendix, or as supplemental material.
% \end{itemize}
% \item {\bf Experiment Statistical Significance}
% \item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
% \item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
% \item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
% \item The assumptions made should be given (e.g., Normally distributed errors).
% \item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
% \item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
% \item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
% \item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
% \end{itemize}
% \item {\bf Experiments Compute Resources}
% \item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
% \item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
% \item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
% \end{itemize}
% \item {\bf Code Of Ethics}
% \item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
% \item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
% \item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
% \end{itemize}
% \item {\bf Broader Impacts}
% \item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that there is no societal impact of the work performed.
% \item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
% \item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
% \item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
% \item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
% \item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
% \end{itemize}
% \item {\bf Safeguards}
% \item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper poses no such risks.
% \item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
% \item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
% \item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
% \end{itemize}
% \item {\bf Licenses for existing assets}
% \item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not use existing assets.
% \item The authors should cite the original paper that produced the code package or dataset.
% \item The authors should state which version of the asset is used and, if possible, include a URL.
% \item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
% \item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
% \item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
% \item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
% \item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
% \end{itemize}
% \item {\bf New Assets}
% \item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not release new assets.
% \item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
% \item The paper should discuss whether and how consent was obtained from people whose asset is used.
% \item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
% \end{itemize}
% \item {\bf Crowdsourcing and Research with Human Subjects}
% \item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
% \item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
% \end{itemize}
% \item {\bf Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects}
% \item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
% \item[] Answer: \answerTODO{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: \justificationTODO{}
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
% \item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
% \item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
% \end{itemize}
% \end{enumerate}
\end{document}
\begin{thebibliography}{10}
\bibitem{agostinelli2014learning}
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi.
\newblock Learning activation functions to improve deep neural networks.
\newblock {\em arXiv preprint arXiv:1412.6830}, 2014.
\bibitem{ba2016layer}
Jimmy~Lei Ba, Jamie~Ryan Kiros, and Geoffrey~E Hinton.
\newblock Layer normalization.
\newblock {\em arXiv preprint arXiv:1607.06450}, 2016.
\bibitem{basirat2020relu}
Mina Basirat and PETER ROTH.
\newblock L* relu: piece-wise linear activation functions for deep fine-grained visual categorization.
\newblock In {\em Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}, pages 1218--1227, 2020.
\bibitem{bungert2021clip}
Leon Bungert, Ren{\'e} Raab, Tim Roith, Leo Schwinn, and Daniel Tenbrinck.
\newblock Clip: Cheap lipschitz training of neural networks.
\newblock In {\em International Conference on Scale Space and Variational Methods in Computer Vision}, pages 307--319. Springer, 2021.
\bibitem{clevert2020fast}
Djork-Arn{\'e} Clevert, Thomas Unterthiner, and Sepp Hochreiter.
\newblock Fast and accurate deep network learning by exponential linear units (elus). arxiv 2015.
\newblock {\em arXiv preprint arXiv:1511.07289}, 2020.
\bibitem{diederik2014adam}
P~Kingma Diederik.
\newblock Adam: A method for stochastic optimization.
\newblock {\em (No Title)}, 2014.
\bibitem{dubey2022activation}
Shiv~Ram Dubey, Satish~Kumar Singh, and Bidyut~Baran Chaudhuri.
\newblock Activation functions in deep learning: A comprehensive survey and benchmark.
\newblock {\em Neurocomputing}, 503:92--108, 2022.
\bibitem{eger2019time}
Steffen Eger, Paul Youssef, and Iryna Gurevych.
\newblock Is it time to swish? comparing deep learning activation functions across nlp tasks.
\newblock {\em arXiv preprint arXiv:1901.02671}, 2019.
\bibitem{gu2019fast}
Shuhang Gu, Wen Li, Luc~Van Gool, and Radu Timofte.
\newblock Fast image restoration with multi-bin trainable linear units.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 4190--4199, 2019.
\bibitem{he2015delving}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.
\newblock In {\em Proceedings of the IEEE international conference on computer vision}, pages 1026--1034, 2015.
\bibitem{he2016identity}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Identity mappings in deep residual networks.
\newblock In {\em Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14}, pages 630--645. Springer, 2016.
\bibitem{hendrycks2016gaussian}
Dan Hendrycks and Kevin Gimpel.
\newblock Gaussian error linear units (gelus).
\newblock {\em arXiv preprint arXiv:1606.08415}, 2016.
\bibitem{ioffe2015batch}
Sergey Ioffe and Christian Szegedy.
\newblock Batch normalization: Accelerating deep network training by reducing internal covariate shift.
\newblock In {\em International conference on machine learning}, pages 448--456. pmlr, 2015.
\bibitem{jiang2022delve}
Ziping Jiang, Yunpeng Wang, Chang-Tsun Li, Plamen Angelov, and Richard Jiang.
\newblock Delve into activations: Towards understanding dying neuron.
\newblock {\em IEEE Transactions on Artificial Intelligence}, 2022.
\bibitem{klambauer2017self}
G{\"u}nter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.
\newblock Self-normalizing neural networks.
\newblock {\em Advances in neural information processing systems}, 30, 2017.
\bibitem{li2018adaptive}
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu.
\newblock Adaptive batch normalization for practical domain adaptation.
\newblock {\em Pattern Recognition}, 80:109--117, 2018.
\bibitem{liu2019natural}
Yang Liu, Jianpeng Zhang, Chao Gao, Jinghua Qu, and Lixin Ji.
\newblock Natural-logarithm-rectified activation function in convolutional neural networks.
\newblock In {\em 2019 IEEE 5th International Conference on Computer and Communications (ICCC)}, pages 2000--2008. IEEE, 2019.
\bibitem{luo2019adaptive}
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu~Sun.
\newblock Adaptive gradient methods with dynamic bound of learning rate.
\newblock {\em arXiv preprint arXiv:1902.09843}, 2019.
\bibitem{pascanu2013difficulty}
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.
\newblock On the difficulty of training recurrent neural networks.
\newblock In {\em International conference on machine learning}, pages 1310--1318. Pmlr, 2013.
\bibitem{ramachandran2017searching}
Prajit Ramachandran, Barret Zoph, and Quoc~V Le.
\newblock Searching for activation functions.
\newblock {\em arXiv preprint arXiv:1710.05941}, 2017.
\bibitem{sharma2018analysis}
Neha Sharma, Vibhor Jain, and Anju Mishra.
\newblock An analysis of convolutional neural networks for image classification.
\newblock {\em Procedia computer science}, 132:377--384, 2018.
\bibitem{shi2016real}
Wenzhe Shi, Jose Caballero, Ferenc Husz{\'a}r, Johannes Totz, Andrew~P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.
\newblock Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.
\newblock In {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, pages 1874--1883, 2016.
\bibitem{srivastava2014dropout}
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.
\newblock Dropout: a simple way to prevent neural networks from overfitting.
\newblock {\em The journal of machine learning research}, 15(1):1929--1958, 2014.
\bibitem{sutskever2013importance}
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.
\newblock On the importance of initialization and momentum in deep learning.
\newblock In {\em International conference on machine learning}, pages 1139--1147. PMLR, 2013.
\bibitem{trottier2017parametric}
Ludovic Trottier, Philippe Giguere, and Brahim Chaib-Draa.
\newblock Parametric exponential linear unit for deep convolutional neural networks.
\newblock In {\em 2017 16th IEEE international conference on machine learning and applications (ICMLA)}, pages 207--214. IEEE, 2017.
\bibitem{vercellino2017hyperactivations}
Conner~Joseph Vercellino and William~Yang Wang.
\newblock Hyperactivations for activation function exploration.
\newblock In {\em 31st Conference on Neural Information Processing Systems (NIPS 2017), Workshop on Meta-learning. Long Beach, USA}, 2017.
\bibitem{wu2018group}
Yuxin Wu and Kaiming He.
\newblock Group normalization.
\newblock In {\em Proceedings of the European conference on computer vision (ECCV)}, pages 3--19, 2018.
\bibitem{zhuang2020adabelief}
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar~C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan.
\newblock Adabelief optimizer: Adapting stepsizes by the belief in observed gradients.
\newblock {\em Advances in neural information processing systems}, 33:18795--18806, 2020.
\bibitem{zoph2016neural}
Barret Zoph and Quoc~V Le.
\newblock Neural architecture search with reinforcement learning.
\newblock {\em arXiv preprint arXiv:1611.01578}, 2016.
\end{thebibliography}