Diff-E: Diffusion-based Learning for Decoding Imagined Speech EEG
Extracted source text · arXiv 2307.14389
\documentclass{INTERSPEECH2023}
% 2023-01-06 modified by Simon King (Simon.King@ed.ac.uk)
% **************************************
% * DOUBLE-BLIND REVIEW SETTINGS *
% **************************************
% Comment out \interspeechcameraready when submitting the
% paper for review.
% If your paper is accepted, uncomment this to produce the
% 'camera ready' version to submit for publication.
\interspeechcameraready
\usepackage{amsmath}
% \usepackage{courier}
% \usepackage{breakurl}
%\usepackage{url}
% \usepackage{hyperref}
% \hypersetup{
% colorlinks=true,
% linkcolor=blue,
% filecolor=magenta,
% urlcolor=cyan,
% pdftitle={Overleaf Example},
% pdfpagemode=FullScreen,
% }
% \urlstyle{same}
% **************************************
% * *
% * STOP ! DO NOT DELETE ! *
% * READ THIS FIRST *
% * *
% * This template also includes *
% * important INSTRUCTIONS that you *
% * must follow when preparing your *
% * paper. Read it BEFORE replacing *
% * the content with your own work. *
% **************************************
\title{Diff-E: Diffusion-based Learning for Decoding Imagined Speech EEG}
\name{Soowon Kim$^1$, Young-Eun Lee$^2$, Seo-Hyun Lee$^2$, Seong-Whan Lee$^1$}
%The maximum number of authors in the author list is 20. If the number of contributing authors is more than this, they should be listed in a footnote or the acknowledgment section.
\address{$^1$Department of Artificial Intelligence, Korea University, Republic of Korea \\
$^2$Department of Brain and Cognitive Engineering, Korea University, Republic of Korea}
\email{soowon\_kim@korea.ac.kr, ye\_lee@korea.ac.kr, seohyunlee@korea.ac.kr, sw.lee@korea.ac.kr}
\begin{document}
\maketitle
\begin{abstract}
Decoding EEG signals for imagined speech is a challenging task due to the high-dimensional nature of the data and low signal-to-noise ratio. In recent years, denoising diffusion probabilistic models (DDPMs) have emerged as promising approaches for representation learning in various domains. Our study proposes a novel method for decoding EEG signals for imagined speech using DDPMs and a conditional autoencoder named Diff-E. Results indicate that Diff-E significantly improves the accuracy of decoding EEG signals for imagined speech compared to traditional machine learning techniques and baseline models. Our findings suggest that DDPMs can be an effective tool for EEG signal decoding, with potential implications for the development of brain-computer interfaces that enable communication through imagined speech.
\end{abstract}
\noindent\textbf{Index Terms}: silent communication, speech recognition, electroencephalography, imagined speech, brain-computer interface
\section{Introduction}
Speech is a crucial mode of communication for humans, allowing us to convey complex ideas and thoughts to others through sound patterns \cite{lieberman1991uniquely, peters2012speaking, lieberman2007evolution}. It is a fundamental part of our social and cultural interactions, enabling us to build relationships, share information, and express emotions. Speech is a highly complex process that involves the coordinated activity of various brain regions and muscles, including the larynx, tongue, lips, and jaw. Unfortunately, several individuals cannot speak due to physical limitations such as locked-in syndrome, a condition where they are fully conscious but unable to move or communicate verbally \cite{laureys2005locked, houde2011speech}.
The inability to speak can significantly impact an individual's quality of life, as it limits their ability to interact with others and express themselves effectively. Therefore, developing effective methods for restoring communication in individuals with locked-in syndrome and other speech impairments is a critical area of research. In this context, our research explores decoding brain signals as a potential method for enabling silent communication \cite{chaudhary2016brain, birbaumer2008brain, chaudhary2017brain}.
% The inability to speak significantly impacts an individual's quality of life, restricting their capacity for social interaction and self-expression. Hence, the quest for effective methods to restore communication in individuals with conditions such as locked-in syndrome is of paramount importance. In this context, our research explores decoding brain signals as a potential method for enabling silent communication. \cite{chaudhary2016brain, birbaumer2008brain, chaudhary2017brain}.
% The inability to speak significantly impacts an individual's quality of life, restricting their capacity for social interaction and self-expression. Hence, the quest for effective methods to restore communication in individuals with conditions such as locked-in syndrome is of paramount importance. In this context, our research explores decoding brain signals as a potential method for enabling silent communication.
Electroencephalography (EEG) is a non-invasive technique for measuring the brain's electrical activity. EEG signals are acquired by placing electrodes on the scalp, which detect the electrical activity generated by the brain's neurons. EEG signals have been widely used in neuroscience research to study brain function, communication, and medical applications such as diagnosing neurological disorders. In recent years, EEG decoding has gained increasing attention to translate EEG signals into meaningful information, such as speech or attention detection \cite{cai2021eeg, su2022stanet, lee2023towards, lee2022eeg}.
% EEG decoding typically involves using machine learning algorithms to decode the EEG signal and extract features corresponding to specific cognitive or motor processes. These features can then be used to control devices, such as prosthetics or assistive technology, or to provide insight into cognitive or speech-related functions \cite{bang2023motion}.
EEG decoding for speech processing is a particularly challenging task, as it involves decoding complex and highly variable neural patterns associated with speech production and perception \cite{li2021biologically}. % However, turning these EEG signals into comprehensible information such as speech or attention detection— a process known as EEG decoding—poses considerable challenges. This is particularly true for speech processing, where decoding is akin to deciphering complex and highly variable neural patterns associated with speech production and perception
Moreover, EEG signals are often contaminated with noise and artifacts, which can affect the accuracy of the decoding process \cite{lee2020real}. Therefore, developing accurate and robust methods for decoding EEG signals associated with imagined speech is an active area of research, with potential applications in speech rehabilitation, human-computer interaction, and other related fields.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth,height=\textheight,keepaspectratio]{forward_process.pdf}
\caption{ The figure illustrates the forward diffusion process applied to a section of the recorded EEG signal captured from Broca's area, FT7, of Subject 1.}
\label{fig:diffusion_forward}
\end{figure}
% Diffusion models have emerged as a powerful tool for learning representations of complex data, particularly for modeling high-dimensional data
Denoising diffusion probabilistic models (DDPMs) have become a powerful tool for identifying sophisticated patterns within complex data, especially when dealing with high-dimensional data \cite{ho2020denoising, song2020score, dhariwal2021diffusion}. For DDPMs, the forward diffusion process introduces noise to the original data, transforming it into a corrupted version. This process is modeled by adding Gaussian noise at each time step, effectively simulating a random walk in the latent space. The aim of the denoising diffusion model is then to reverse this process, i.e., to denoise the corrupted data and recover the original signal.
% Diffusion models use a stochastic diffusion process to learn the probability distribution of the data.
% This process involves iteratively transforming the data by gradually adding noise and smoothing the resulting samples, then the model aims to understand the inverse of a stochastic process that gradually erodes information, thus capturing the temporal dependencies and correlations in the data.
In recent years, DDPMs have gained attention as a promising approach for time-series data processing and analysis as well, including audio and video \cite{tashiro2021csdi, rasul2021autoregressive}. In the context of EEG decoding, DDPMs may offer a promising approach for capturing the complex relationships between neural activity and language processing during imagined speech tasks. This can enable accurate decoding of EEG signals associated with imagined speech, which has potential applications in various fields such as speech rehabilitation and neuroscience.
DDPMs have also been employed in representation learning. Preechakul et al. \cite{preechakul2022diffusion} demonstrated that diffusion autoencoders can yield a more interpretable and decodable latent data representation than conventional autoencoders. Zhang et al. \cite{zhang2022unsupervised} displayed the effectiveness of pre-trained diffusion models using high-dimensional data representations. Their approach involves a trainable model that predicts a mean shift based on the encoded representation. It is trained to bridge as much of the gap as possible, a gap that emerges during the forward process of DDPMs. Consequently, the encoder is compelled to extract as much information as it can from images to assist with this bridging process.
% In this study, building upon prior research, we introduce a method to utilize the powerful learning capacity of Denoising Diffusion Probabilistic Models (DDPMs) in tandem with a Conditional Autoencoder (CAE) to learn and represent intricate patterns inherent in EEG signals. The primary objective of employing the CAE is to discern and learn from the meaningful features that might not be fully captured or might be lost during the forward process of DDPMs. Additionally, to enhance decoding performance, a classifier is incorporated and jointly trained with the CAE. To the best of our knowledge, this proposed approach represents the first attempt to utilize diffusion models to decode EEG signals associated with imagined speech. 일단 다른버전으로 그냥 써봄. 위에 글 너무 길면 일캐.
In this study, building upon prior research, we present a novel approach to decode EEG signals using DDPMs and a Conditional Autoencoder (CAE). The CAE aids in learning meaningful features potentially lost during the forward process in DDPMs. Furthermore, we've integrated a jointly-trained classifier to improve decoding performance. To the best of our knowledge, this proposed approach represents the first attempt to utilize diffusion models to decode EEG signals associated with imagined speech.
% In this study, especially in light of the previous work, we describe a method for learning meaningful representations of EEG signals using DDPMs and a conditional autoencoder (CAE). The aim of the CAE is to learn the information loss that occurs during the forward process of DDPMs. In addition, a classifier is jointly trained with the CAE to further improve the classification performance. To the best of our knowledge, this proposed approach represents the first attempt to utilize diffusion models to decode EEG signals associated with imagined speech.
\section{Related Works}
Decoding EEG signals using deep learning approaches is a challenging problem due to various factors, including the scarcity of data, poor signal-to-noise ratio, and high inter- and intra-individual variability \cite{tayeb2019validating}. Despite these challenges, several studies have explored different EEG decoding techniques for various applications, including speech decoding.
% One notable study by Schirrmeister et al. \cite{schirrmeister2017deep} developed DeepConvNets for end-to-end learning of EEG signals recorded in human subjects. The authors utilized recent advances from the machine learning field, such as batch normalization and exponential linear units, and a cropped training strategy to improve DeepConvNet decoding performance, achieving accuracy levels similar to filter bank common spatial patterns (FBCSP) algorithms. The authors also used novel visualization techniques to show that ConvNets learn spectral power modulations in alpha, beta, and high gamma frequencies, which can be useful for spatially localizing informative brain regions related to imagined or executed tasks from raw EEG data. Another study by \cite{lawhern2018eegnet} proposed EEGNet, a compact CNN architecture specifically designed for EEG classification tasks. EEGNet uses depthwise and separable convolutions, which can extract features tailored to the unique characteristics of EEG control signals. The model has been tested on four BCI paradigms and shown to accurately classify EEG signals while being as compact as possible with respect to the number of parameters in the model.
Schirrmeister et al. \cite{schirrmeister2017deep} developed DeepConvNets for end-to-end learning of EEG signals in human subjects. They utilized machine learning techniques such as batch normalization and exponential linear units and a cropped training strategy to improve decoding performance, achieving accuracy levels similar to filter bank common spatial patterns algorithms. Another study by Lawhern et al. \cite{lawhern2018eegnet} proposed EEGNet, a compact CNN architecture specifically designed for EEG classification tasks. EEGNet uses depthwise and separable convolutions, which can extract features tailored to the unique characteristics of EEG control signals. The model has been tested on four BCI paradigms and shown to accurately classify EEG signals while being compact in terms of the number of parameters in the model.
Lee et al. \cite{lee2020neural} investigated the underlying characteristics affecting the decoding performance of two emerging BCI paradigms: imagined speech and visual imagery. The authors analyzed EEG signals filtered in six frequency ranges for 13 class classifications, including a rest class. They identified cortical regions relevant to both paradigms using semantic levels of word properties according to their concepts. The study showed high accuracy rates across all classes and multi-class scalability in both paradigms, providing crucial information for improving the efficiency of BCI applications.
Diffusion-based methods for time-series data have been extensively researched recently. One such method proposed in \cite{alcaraz2022diffusion} is based on a structured state space model that incorporates a diffusion process to capture the underlying dynamics of the data. This method is able to impute and forecast missing values in time series data by utilizing the estimated diffusion process. The authors demonstrate the effectiveness of their approach on various real-world datasets and show that it outperforms existing methods for time-series imputation and forecasting. Jeong et al. \cite{jeong2021diff} propose a method for improving the quality of synthetic speech generated by TTS systems. The proposed method utilizes DDPMS to remove noise from the intermediate representations of speech generated by TTS systems. By reducing noise in the intermediate representations, the proposed method is able to produce clearer and more natural-sounding speech. The authors evaluate the effectiveness of their approach on several real-world datasets and demonstrate that it outperforms existing methods for improving the quality of synthetic speech.
% Overall, these studies demonstrate the potential of deep learning methods in improving the accuracy and efficiency of EEG decoding tasks. However, challenges remain in decoding EEG signals, and continued research and development are necessary to address these challenges and improve the reliability and applicability of EEG decoding methods.
\section{Materials and Methods}
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth,height=\textheight,keepaspectratio]{DiffE.pdf}
\caption{A flowchart of Diff-E for EEG signal decoding. The nodes labeled as \texttt{Conv1D}, \texttt{GN}, and \texttt{Linear} represent one-dimensional convolutional layer, group normalization, and linear layer, respectively. The node labeled as \texttt{ELU} represents a nonlinear activation function. Adaptive average pooling layer, \texttt{AdaPool} is used to make input for the classifier, \texttt{FC}.}
\label{fig:method}
\end{figure}
% The input EEG signal is first processed by the DDPM to generate a noisy signal. The downsampling layer output from DDPM and the original signal $\mathbf{x}_0$ is then passed to the CAE, which attempts to identify and rectify the errors introduced by the DDPM during the forward pass. The output of the encoder in CAE, $\mathbf{z}$, is a compact representation of the data, which is passed to the linear classifier. The linear classifier is jointly trained with the CAE to further improve the decoding performance. The final output of the proposed method is the classification of the input EEG signal.
\subsection{Denoising Diffusion Models}
DDPMs are a type of machine learning model that can learn complex probability distributions over data. The "forward process" in DDPMs is determined by a fixed Markov chain that progressively introduces Gaussian noise to the data. The forward process in DDPMs starts with a probability distribution called $q(\mathbf{x}_0)$, which represents the uncorrupted original data. This distribution is then transformed iteratively using a sequence of Markov diffusion kernels called $q(\mathbf{x}_t|\mathbf{x}_{t-1})$, which are Gaussian with a fixed variance schedule $\{\beta_t\}^T_{t=1}$. The process can be expressed by:
\begin{equation}
q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\mathbf{I})
\end{equation}
\hspace{0.5cm}
\begin{equation}
q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^{T}q(\mathbf{x}_t|\mathbf{x}_{t-1})
\end{equation}
Diffusion process with Gaussian noise can corrupt the original data with arbitrary stage, $t$ with the notation $\alpha_t=1-\beta_t$ and $\Bar{\alpha}_t=\prod_{s=1}^{t}\alpha_s$:
\begin{equation}
q(\mathbf{x}_t|\mathbf{x}_0)=\mathcal{N}(\mathbf{x}_t;\sqrt{\Bar{\alpha}_t}\mathbf{x}_0, (1-\Bar{\alpha}_t) \mathbf{I})
\end{equation}
Ho et al. \cite{ho2020denoising} proposed a method to train a model that takes a noisy sample $\mathbf{x}t$ and predicts its noise by training a network $\epsilon\theta(\mathbf{x}_t,t)$. In contrast, our study trains Diff-E to predict the uncorrupted original signal, $\mathbf{x}_0$, instead of predicting the injected noise:
\begin{equation}
\mathcal{L}_{\text{DDPM}}(\theta) = ||\mathbf{x}_0 - \hat{\mathbf{x}}_\theta(\mathbf{x}_t,t)||
\label{eq:DDPM}
\end{equation}
Here, $\theta$ are the DDPM parameters, and $\hat{\mathbf{x}}_\theta(\mathbf{x}_t,t)$ is the prediction of the DDPM. We randomly sample timestep $t$ from a uniform distribution, $\mathcal{U}(1, T)$. The objective function encourages the model to denoise the noisy input and produce output that is close to the original input signal. We have adopted a time-conditional UNet architecture \cite{ronneberger2015u} as described in Fig. \ref{fig:method}, similar to the model used in the original paper \cite{ho2020denoising}, with modifications to adapt it for EEG data.
\subsection{Conditional Autoencoder}
Similar to the work \cite{zhang2022unsupervised}, the CAE aims to derive meaningful representations of EEG signals by leveraging the errors that occur during the reconstruction process by the DDPM. The forward pass of the DDPM causes information loss, which the CAE attempts to compensate for by identifying and rectifying these errors. This enables the CAE to generate more accurate representations of the original EEG signals. Specifically, we employ the objective function for the CAE as follows:
\begin{equation}
\mathcal{L}_{\text{CAE}}(\psi, \phi) = ||\mathcal{L}_{\text{DDPM}}(\theta) -\mathcal{D}_\psi(\mathcal{E}_\phi(\mathbf{x}_0),\hat{\mathbf{x}}_\theta(\mathbf{x}_t, t))||,
\label{eq:CAE}
\end{equation}
The CAE is composed of an encoder network, denoted by $\mathcal{E}_\phi$, and a decoder network, denoted by $\mathcal{D}_\psi$, which is similar in structure to the DDPM. However, $\mathcal{D}_\psi$ receives skip connections not from the outputs of $\mathcal{E}_\phi$, but from the DDPM layers. This enables $\mathcal{D}_\psi$ to be implicitly conditioned on the corruption stage of the forward process of the DDPM, as shown in Fig. \ref{fig:method} as \textit{dashed arrows}. Furthermore, to improve the reconstruction of $\mathcal{L}_{\text{DDPM}}$, the original signal, $\mathbf{x}_0$, and the output of the DDPM, $\hat{\mathbf{x}}_\theta(\mathbf{x}_t,t)$, are also skip-connected as inputs to the last layer of $\mathcal{D}_\psi$ as depicted in Fig. \ref{fig:method} as \textit{thin dashed arrows}.
\subsection{Classifier}
After $\mathcal{E}_\phi$ processes the data, the output is transformed into a compact representation, $\mathbf{z}$, by collapsing the time dimension into a single dimension. This is achieved by applying an adaptive average pooling layer, which effectively creates a one-dimensional latent vector. The compressed representation is then fed into the linear classifier, $\mathcal{C}_\rho$, which is jointly trained with the CAE to further separate the representations of each class and perform the classification task. The dimension of $\mathbf{z}$ is constant at $256$ throughout the experiment. To include the classification loss in the objective function of the CAE, we modified it as follows to become the overall Diff-E objective:
\begin{equation}
\begin{split}
\mathcal{L}_{\text{Diff-E}}(\psi, \phi, \rho) &= ||\mathcal{L}_{\text{DDPM}}(\theta) -\mathcal{D}_\psi(\mathcal{E}_\phi(\mathbf{x}_0),\hat{\mathbf{x}}_\theta(\mathbf{x}_t, t))||\\
&+\alpha||\hat{y}-y||_2
\end{split}
\label{eq:Diff-E}
\end{equation}
where $\hat{y}=\mathcal{C}_\rho(\mathbf{z})$. Here, $\rho$ is adjustable parameters for $\mathcal{C}_\rho$, and $y$ is the true label of the input signal. The hyperparameter $\alpha$ governs the relative importance of the reconstruction loss and the classification loss. $0.1$ is chosen for the value of $\alpha$ throughout the experiment. During the inference, only $\mathcal{E}_\phi$ and $\mathcal{C}_\rho$ are used to classify the signals. Specifically, the predicted label is obtained as $\hat{y}=\mathcal{E}_\phi(\mathcal{C}_\rho(\mathbf{x}_0))$.
In order to evaluate the effectiveness of Diff-E, it is compared to other existing methods that have demonstrated their applicability to decoding EEG signals in various paradigms, such as motor imagery and event-related potentials. This comparison is conducted to assess the performance of Diff-E and to determine its suitability for imagined speech EEG signal decoding applications.
\subsection{Model Implementation Details}
The DDPM and CAE architectures in our study were designed with each layer comprising a one-dimensional convolution (\texttt{Conv1D}), group normalization (\texttt{GN}), and activation function (\texttt{ELU}). In $\mathcal{E}_\phi$, adaptive average pooling (\texttt{AdaPool}) is applied to transform the output into a compressed representation, $\mathbf{z}$. The total number of trainable parameters for both DDPM and CAE was approximately 300,000, while the linear classifier had 400,000 trainable parameters.
We trained the models using the RMSProp optimizer with a fixed random seed of 42 and a cyclic learning rate scheduler developed by Smith \cite{smith2017cyclical}. The base learning rate was set to $9*10^{-5}$, and the maximum learning rate was set to $1.5*10^{-3}$. The training process was performed over a total of 500 epochs. During training, we used L1 loss for both the DDPM and CAE components of the model. We also used the mean squared error loss against one-hot encoded labels for the linear classifier classification task. To evaluate the performance of the model, we set aside 20\% of the data for testing with a fixed random seed. Our implementation is available at \url{https://github.com/yorgoon/DiffE/}.
\subsection{Dataset}
\subsubsection{Data Description}
The data utilized in this study was obtained from a prior study conducted by Lee et al. \cite{lee2020neural}. The study involved recruiting twenty-two healthy participants, of whom fifteen were male, with a mean age of 24.68 ± 2.15. None of the participants had a history of neurological disease or language disorder, and they did not have any hearing or visual impairments. Additionally, they refrained from taking drugs for twelve hours before the session. All participants had received high-quality English education for more than 15 years. In the imagined speech task, twenty-two subjects were instructed to imagine saying twelve different words or sentences, such as ``ambulance," ``clock," ``hello," ``help me," ``light," ``pain," ``stop," ``thank you," ``toilet," ``TV," ``water," and ``yes," as well as a resting state, resulting in a total of thirteen classes that a communication board commonly used in hospitals for patients with paralysis or aphasia. These words are considered to be essential for patient communication. The researchers used a 64-channel EEG cap with active Ag/AgCl electrodes that followed the international 10-10 system to record the EEG signals. The FCz and FPz channels were set as the reference and ground electrodes, respectively. Brain Vision/Recorder software (BrainProduct GmbH, Germany) was used to collect the EEG signals, which were then operated using MATLAB 2018a software. The researchers ensured that the impedance of all electrodes was maintained below $10k\Omega$. The researchers randomly presented twenty-two blocks of twelve words and a rest class. Each of the twenty-two participants contributed one thousand three hundred samples, consisting of one hundred samples per category. The study was approved by the Korea University Institutional Review Board [KUIRB-2019-0143-01] and was conducted in accordance with the Declaration of Helsinki.
% 오늘 교수님 탠션 뭐지 여튼 이렇게 해서 교수님한태 가져갈까요 세미나 끝나고 갖고 오라그래서 ㅎㅎ 오케이~ ㅋ캐캐이~ 우캐캐이~ 끼끼~ ㅋㅋㅋㅋㅋ리얼 무엇 ㅇㅇㅇ 괜찮아보여요 일단 낄낄끼 우끼끼끼끼끼ㅣ 원숭이 원숭원 이원승씨 모릅니까 옜날 개그맨ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ다른 사람들을 고문하는거랰ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ 쓸데없는
\subsubsection{Preprocessing}
In this study, we employed several preprocessing steps to ensure the quality and accuracy of the EEG data. The first step was to apply band-pass filtering to the signals between 0.5 and 125Hz, with additional notch filtering at 60 and 120Hz to eliminate power line interference. The data was then re-referenced using a common average reference method to reduce the impact of any noise present in the data. To eliminate ocular and muscular artifacts that can be caused by movement or sounds, we used automatic electrooculography and electromyography removal methods. After removing the artifacts, the EEG signals were selected in the high-gamma frequency band for training the model and data analysis. The dataset was then epoched into 2-second segments, with baseline correction applied 500ms before the task. All of the preprocessing steps were carried out using MATLAB-based tools, including the OpenBMI Toolbox \cite{leeMH2019eeg,lee2021decoding}, BBCI Toolbox \cite{krepki2007berlin}, and EEGLAB \cite{Delorme2004EEGLAB}.
\section{Results and Discussion}
\subsection{Performance Comparison}
% In this study, we compared the performance of our proposed Diff-E method with three other methods, namely DeepConvNet \cite{schirrmeister2017deep}, EEGNet \cite{lawhern2018eegnet}, and Lee et al. \cite{lee2020neural}, for EEG signal decoding. The performance of each method was evaluated across all subjects, and the results are presented in Table \ref{tab:result}. The table reports the accuracy and area under the curve (AUC) for each method. Diff-E significantly outperformed all other methods, including DeepConvNet, EEGNet, and Lee et al. \cite{lee2020neural}, in both accuracy and AUC. The average accuracy of Diff-E was 60.63\%, with a standard deviation of 8.03\%. The average AUC of Diff-E was 90.39\%, with a standard deviation of 3.48\%. Compared to the other methods, DeepConvNet had an average accuracy of 21.00\%, EEGNet had an average accuracy of 32.78\%, and Lee et al. had an average accuracy of 46.27\%. The results indicate that our proposed method is significantly more effective in decoding imagined speech EEG signals than baseline methods.
In this study, we compared the performance of our proposed Diff-E method with three other methods, DeepConvNet \cite{schirrmeister2017deep}, EEGNet \cite{lawhern2018eegnet}, and Lee et al. \cite{lee2020neural}, for EEG signal decoding. The results are presented in Table \ref{tab:result}. Diff-E outperformed all other methods, including DeepConvNet, EEGNet, and Lee et al., in both accuracy and area under the curve (AUC). The average accuracy of Diff-E was 60.63\%, with a standard deviation of 8.03\%, and the average AUC was 90.39\%, with a standard deviation of 3.48\%. Notably, Diff-E achieved significantly higher accuracy and AUC than the other methods. The average accuracy of DeepConvNet, EEGNet, and Lee et al. were 21.00\%, 32.78\%, and 46.27\%, respectively. The average AUC of DeepConvNet, EEGNet, and Lee et al. were 69.55\%, 78.07\%, and 79.97\%, respectively. These findings highlight the superior performance of our proposed Diff-E method for EEG signal decoding.
Our study's results were unexpected since commonly used, and well-known methods such as EEGNet and DeepConvNet for EEG decoding underperformed compared to a traditional machine-learning method, such as a common spatial pattern-support vector machine method used by Lee et al. \cite{lee2020neural}. These methods are frequently used for decoding EEG signals from various paradigms, including motor imagery and event-related potentials. Our findings emphasize the importance of carefully selecting the appropriate model architecture based on the task at hand, the EEG paradigms used, and other relevant factors.
% However, other studies have attempted to use deep learning methods for decoding EEG signals from imagined speech \cite{cooney2019optimizing, li2021decoding}. Nevertheless, their methods have been limited in their scope, as \cite{li2021decoding} trained the model with only 1,600 samples in total and \cite{cooney2019optimizing} had to classify only five classes. Our proposed method, which is based on a diffusion model, has the potential to overcome these limitations and accelerate the development of effective and modern deep-learning models for EEG signal decoding. This could revolutionize the field of brain-computer interfaces and enable communication through imagined speech.
% Recent studies such as \cite{cooney2019optimizing, li2021decoding} have attempted to utilize deep learning methods for decoding EEG signals from imagined speech. However, these studies have been limited in their scope. For instance, \cite{li2021decoding} trained their model with only 1,600 samples in total, which may not be enough for optimizing large deep-learning models effectively. Meanwhile, \cite{cooney2019optimizing} had to classify only five classes, which may not be representative of the complexity of imagined speech decoding. In contrast, our proposed method based on a diffusion model has the potential to overcome these limitations and accelerate the development of effective and modern deep-learning models for EEG signal decoding.
% 이 명령은 표의 상하 여백을 숫자로 조정한다.
% 이 명령은 표의 좌우 여백을 숫자로 조정한다.
\begin{table}[t]
\centering
% \renewcommand{\arraystretch}{1}
\renewcommand{\tabcolsep}{3mm}
\caption{Accuracy and AUC scores for imagined speech classification}
\label{tab:result}
% \resizebox{\columnwidth}{!}{}
{
\begin{tabular}{lcc}
\hline
\textbf{Subject} & \textbf{Accuracy (\%)} & \textbf{AUC (\%)}\\
\hline
\textbf{DeepConvNet} & 21.00 $\pm$ 4.27 & 69.55 $\pm$ 3.51 \\
\textbf{EEGNet} & 32.78 $\pm$ 4.98 & 78.07 $\pm$ 4.27 \\
\textbf{Lee et al.\cite{lee2020neural}} & 46.27 $\pm$ 6.79 & 79.97 $\pm$ 4.83 \\
\textbf{Diff-E} & \textbf{60.63 $\pm$ 8.03} & \textbf{90.39 $\pm$ 3.48} \\
\hline
\end{tabular}
}
\end{table}
\begin{table}[t]
% \renewcommand{\arraystretch}{1.2}
\centering
% \renewcommand{\tabcolsep}{3mm}
\caption{Ablation study}
% \resizebox{\columnwidth}{!}{}
{
\begin{tabular}{lcc}
\hline
\textbf{Components} & \textbf{Accuracy (\%)} & \textbf{AUC (\%)}\\
\hline
\textbf{Diff-E} & \textbf{60.63 $\pm$ 8.03} & \textbf{90.39 $\pm$ 3.48} \\
\textbf{w/o DDPM} & 46.59 $\pm$ 8.35 & 86.54 $\pm$ 5.36 \\
\textbf{w/o DDPM \& $\mathcal{D}_\psi$} & 47.01 $\pm$ 7.80 & 46.27 $\pm$ 6.79 \\
\hline
\end{tabular}%
}
\label{tab:ablation}
\end{table}
\subsection{Ablation Study}
To evaluate the impact of each component of our proposed method, we performed an ablation study. The study involved removing the diffusion process and training the CAE's encoder and classifier separately to assess the effects of learning from the DDPM's errors and whether the autoencoder-style approach is advantageous for learning meaningful representations. We aimed to determine the significance of each component of Diff-E. During the training process, the CAE was trained only to reconstruct the original signals, and a linear classifier was used without the use of DDPM. Additionally, we conducted another ablation study solely on the $(\mathcal{E}_\phi,\mathcal{C}_\rho)$ component for the classification task, which is essentially a CNN-based classifier.
As shown in Table \ref{tab:ablation}, the results indicate that Diff-E outperformed the other models, and removing the DDPM or solely using $(\mathcal{E}_\phi,\mathcal{C}_\rho)$ led to a significant decrease in accuracy. These results indicate that without DDPM, CAE alone does not offer any advantage over traditional machine learning techniques.
\subsection{Future Works}
% To improve our proposed method for EEG signal decoding, we aim to address its limitations. Despite our EEG dataset being relatively large, it may still be too small for large deep-learning models to optimize effectively. However, our results show that the diffusion model can still provide useful training signals for learning compact and meaningful representations of the data. We plan to enhance the robustness and applicability of our method by collecting larger datasets and applying our method to different EEG paradigms, such as motor imagery and event-related potentials. Additionally, we aim to explore different generative-based methods to aid in the development of more effective and robust models for EEG decoding. Finally, we plan to investigate methods for end-to-end learning directly from raw EEG signals to reduce the need for extensive pre-processing and enhance the efficiency of our method.
% To enhance the effectiveness and robustness of our proposed method for EEG signal decoding, we intend to address several limitations in our future work. Our method was tested on a non-public dataset, as it had larger and more diverse classes than other public datasets, which usually have only a few classes \cite{coretto2017open}. Although our EEG dataset was relatively large, it may still be inadequate for large deep-learning models to optimize effectively. Nevertheless, our results showed that the diffusion model can still provide valuable training signals for learning meaningful and concise representations of the data, even with limited amounts of data. To improve the applicability of our method, we plan to collect larger datasets with more subjects and apply our method to various EEG paradigms, including motor imagery and event-related potentials. This will offer a more comprehensive evaluation of our method's effectiveness and improve its robustness across different paradigms. Moreover, we aim to explore diverse generative-based methods to gain insight into the advantages and disadvantages of each approach, which will aid in developing more effective and robust models for EEG decoding. Additionally, we plan to investigate methods for end-to-end learning directly from raw EEG signals, potentially reducing the need for extensive pre-processing and enhancing the efficiency of our method. Specifically, we aim to explore how deep-learning models can directly process raw EEG signals and investigate methods for addressing the high-dimensional and noisy nature of EEG data.
To enhance the effectiveness and robustness of our proposed method for EEG signal decoding, we intend to address several limitations in our future work. Our method was tested on a non-public dataset, as it had larger and more diverse classes than other public datasets, which usually have only a few classes \cite{coretto2017open}. Although our EEG dataset was relatively large, it may still be inadequate for large deep-learning models to optimize effectively. To improve the applicability of our method, we plan to collect larger datasets with more subjects and apply our method to various EEG paradigms, including motor imagery and event-related potentials. This will offer a more comprehensive evaluation of our method's effectiveness and improve its robustness across different paradigms. Additionally, we plan to investigate methods for end-to-end learning directly from raw EEG signals, potentially reducing the need for extensive pre-processing and enhancing the efficiency of our method.
\section{Conclusion}
% Our study proposes a novel approach using DDPMs for decoding EEG signals of imagined speech, which has demonstrated promising results. Our findings suggest that DDPMs have the potential to be a useful tool for processing EEG signals and extracting relevant features, allowing deep-learning models to effectively decode imagined speech. Furthermore, our study emphasizes the importance of selecting appropriate model architectures for different EEG paradigms to improve the accuracy and performance of the model. In summary, our approach provides a novel method for decoding imagined speech EEG signals and has the potential to advance the field of brain-computer interfaces for communication and assistive technology. By leveraging DDPMs, we have demonstrated a promising approach for improving the accuracy and robustness of EEG signal decoding for imagined speech.
% Our study proposes a novel approach using DDPMs for decoding EEG signals of imagined speech, which has demonstrated promising results. Our findings suggest that DDPMs have the potential to be a useful tool for processing EEG signals and improving the accuracy and performance of deep-learning models for decoding imagined speech. Furthermore, our study emphasizes the importance of selecting appropriate model architectures for different EEG paradigms to improve their effectiveness. In summary, our approach provides a promising method for decoding imagined speech EEG signals and has the potential to advance the field of brain-computer interfaces for communication and assistive technology. By leveraging DDPMs, we have demonstrated an effective approach for improving the accuracy and robustness of EEG signal decoding for imagined speech.
% Our study has introduced a novel approach that utilizes DDPMs to decode EEG signals of imagined speech, and our results have shown promising outcomes. Our findings suggest that DDPMs can serve as a useful tool for processing EEG signals, and can significantly improve the accuracy and performance of deep-learning models in decoding imagined speech. Moreover, we have highlighted the importance of carefully selecting appropriate model architectures that are tailored to different EEG paradigms to enhance their effectiveness in EEG signal decoding. In summary, our proposed approach provides a promising method for accurately decoding EEG signals related to imagined speech, which has the potential to revolutionize the field of brain-computer interfaces for communication and assistive technology. By leveraging the advantages of DDPMs, our study has shown an effective approach to improving the accuracy and robustness of EEG signal decoding for imagined speech.
This work contributes to the growing body of research on deep-learning-based approaches to EEG signal decoding, specifically for the task of imagined speech. The use of DDPMs, a relatively new approach in this field, provides an innovative solution that has demonstrated superior performance compared to traditional methods such as DeepConvNet and EEGNet. The results of our study suggest that DDPMs have the potential to become a valuable tool in EEG signal processing and can be further explored for other applications in this domain. Our study also highlights the importance of selecting appropriate model architectures for different EEG paradigms. This consideration is crucial to achieving high accuracy and performance in EEG signal decoding, as demonstrated by the inferior performance of DeepConvNet and EEGNet in our study. The selection of an appropriate model architecture should take into account various factors, such as the characteristics of the EEG data and the task at hand. Overall, our study provides a promising approach to accurately decoding EEG signals related to imagined speech, which can have significant implications in the development of brain-computer interfaces for communication and assistive technology. This work also contributes to advancing the understanding and utilization of deep-learning-based approaches in EEG signal processing and decoding.
%\section{Acknowledgements}
%\ifinterspeechfinal
% The INTERSPEECH 2023 organizers
%\else
% The authors
%\fi
\section{Acknowledgements}
\thanks{This work was supported by Institute for Information \& Communications Technology Planning \& Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-02068, Artificial Intelligence Innovation Hub; No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University); No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning).}
\bibliographystyle{IEEEtran}
\bibliography{mybib}
\end{document}
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl
\bibitem{lieberman1991uniquely}
P.~Lieberman, \emph{Uniquely human: The evolution of speech, thought, and
selfless behavior}.\hskip 1em plus 0.5em minus 0.4em\relax Harvard University
Press, 1991.
\bibitem{peters2012speaking}
J.~D. Peters, \emph{Speaking into the air: A history of the idea of
communication}.\hskip 1em plus 0.5em minus 0.4em\relax University of Chicago
Press, 2012.
\bibitem{lieberman2007evolution}
P.~Lieberman, ``The evolution of human speech: Its anatomical and neural
bases,'' \emph{Current Anthropology}, vol.~48, no.~1, pp. 39--66, 2007.
\bibitem{laureys2005locked}
S.~Laureys, F.~Pellas, P.~Van~Eeckhout, S.~Ghorbel, C.~Schnakers, F.~Perrin,
J.~Berre, M.-E. Faymonville, K.-H. Pantke, F.~Damas \emph{et~al.}, ``The
locked-in syndrome: what is it like to be conscious but paralyzed and
voiceless?'' \emph{Progress in Brain Research}, vol. 150, pp. 495--611, 2005.
\bibitem{houde2011speech}
J.~F. Houde and S.~S. Nagarajan, ``Speech production as state feedback
control,'' \emph{Frontiers in Human Neuroscience}, vol.~5, p.~82, 2011.
\bibitem{chaudhary2016brain}
U.~Chaudhary, N.~Birbaumer, and A.~Ramos-Murguialday, ``Brain--computer
interfaces for communication and rehabilitation,'' \emph{Nature Reviews
Neurology}, vol.~12, no.~9, pp. 513--525, 2016.
\bibitem{birbaumer2008brain}
N.~Birbaumer, A.~R. Murguialday, and L.~Cohen, ``Brain--computer interface in
paralysis,'' \emph{Current Opinion in Neurology}, vol.~21, no.~6, pp.
634--638, 2008.
\bibitem{chaudhary2017brain}
U.~Chaudhary, B.~Xia, S.~Silvoni, L.~G. Cohen, and N.~Birbaumer,
``Brain--computer interface--based communication in the completely locked-in
state,'' \emph{PLoS Biology}, vol.~15, no.~1, p. e1002593, 2017.
\bibitem{cai2021eeg}
S.~Cai, E.~Su, L.~Xie, and H.~Li, ``{EEG}-based auditory attention detection
via frequency and channel neural attention,'' \emph{IEEE Transactions on
Human-Machine Systems}, vol.~52, no.~2, pp. 256--266, 2021.
\bibitem{su2022stanet}
E.~Su, S.~Cai, L.~Xie, H.~Li, and T.~Schultz, ``{STAnet}: A spatiotemporal
attention network for decoding auditory spatial attention from {EEG},''
\emph{IEEE Transactions on Biomedical Engineering}, vol.~69, no.~7, pp.
2233--2242, 2022.
\bibitem{lee2023towards}
Y.-E. Lee, S.-H. Lee, S.-H. Kim, and S.-W. Lee, ``Towards voice reconstruction
from {EEG} during imagined speech,'' \emph{arXiv preprint arXiv:2301.07173},
2023.
\bibitem{lee2022eeg}
Y.-E. Lee and S.-H. Lee, ``{EEG}-transformer: Self-attention from transformer
architecture for decoding {EEG} of imagined speech,'' \emph{2022 10th
International Winter Conference on Brain-Computer Interface}, pp. 1--4, 2022.
\bibitem{li2021biologically}
P.~Li, S.~Cai, E.~Su, and L.~Xie, ``A biologically inspired attention network
for {EEG}-based auditory attention detection,'' \emph{IEEE Signal Processing
Letters}, vol.~29, pp. 284--288, 2021.
\bibitem{lee2020real}
Y.-E. Lee, N.-S. Kwak, and S.-W. Lee, ``A real-time movement artifact removal
method for ambulatory brain-computer interfaces,'' \emph{IEEE Transactions on
Neural Systems and Rehabilitation Engineering}, vol.~28, no.~12, pp.
2660--2670, 2020.
\bibitem{ho2020denoising}
J.~Ho, A.~Jain, and P.~Abbeel, ``Denoising diffusion probabilistic models,''
\emph{Advances in Neural Information Processing Systems}, vol.~33, pp.
6840--6851, 2020.
\bibitem{song2020score}
Y.~Song, J.~Sohl-Dickstein, D.~P. Kingma, A.~Kumar, S.~Ermon, and B.~Poole,
``Score-based generative modeling through stochastic differential
equations,'' \emph{arXiv preprint arXiv:2011.13456}, 2020.
\bibitem{dhariwal2021diffusion}
P.~Dhariwal and A.~Nichol, ``Diffusion models beat gans on image synthesis,''
\emph{Advances in Neural Information Processing Systems}, vol.~34, pp.
8780--8794, 2021.
\bibitem{tashiro2021csdi}
Y.~Tashiro, J.~Song, Y.~Song, and S.~Ermon, ``{CSDI}: Conditional score-based
diffusion models for probabilistic time series imputation,'' \emph{Advances
in Neural Information Processing Systems}, vol.~34, pp. 24\,804--24\,816,
2021.
\bibitem{rasul2021autoregressive}
K.~Rasul, C.~Seward, I.~Schuster, and R.~Vollgraf, ``Autoregressive denoising
diffusion models for multivariate probabilistic time series forecasting,''
\emph{International Conference on Machine Learning}, pp. 8857--8868, 2021.
\bibitem{preechakul2022diffusion}
K.~Preechakul, N.~Chatthee, S.~Wizadwongsa, and S.~Suwajanakorn, ``Diffusion
autoencoders: Toward a meaningful and decodable representation,''
\emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition}, pp. 10\,619--10\,629, 2022.
\bibitem{zhang2022unsupervised}
Z.~Zhang, Z.~Zhao, and Z.~Lin, ``Unsupervised representation learning from
pre-trained diffusion probabilistic models,'' \emph{Advances in Neural
Information Processing Systems}, vol.~35, pp. 22\,117--22\,130, 2022.
\bibitem{tayeb2019validating}
Z.~Tayeb, J.~Fedjaev, N.~Ghaboosi, C.~Richter, L.~Everding, X.~Qu, Y.~Wu,
G.~Cheng, and J.~Conradt, ``Validating deep neural networks for online
decoding of motor imagery movements from {EEG} signals,'' \emph{Sensors},
vol.~19, no.~1, p. 210, 2019.
\bibitem{schirrmeister2017deep}
R.~T. Schirrmeister, J.~T. Springenberg, L.~D.~J. Fiederer, M.~Glasstetter,
K.~Eggensperger, M.~Tangermann, F.~Hutter, W.~Burgard, and T.~Ball, ``Deep
learning with convolutional neural networks for {EEG} decoding and
visualization,'' \emph{Human Brain Mapping}, vol.~38, no.~11, pp. 5391--5420,
2017.
\bibitem{lawhern2018eegnet}
V.~J. Lawhern, A.~J. Solon, N.~R. Waytowich, S.~M. Gordon, C.~P. Hung, and
B.~J. Lance, ``{EEGNet}: a compact convolutional neural network for
{EEG}-based brain--computer interfaces,'' \emph{Journal of Neural
Engineering}, vol.~15, no.~5, p. 056013, 2018.
\bibitem{lee2020neural}
S.-H. Lee, M.~Lee, and S.-W. Lee, ``Neural decoding of imagined speech and
visual imagery as intuitive paradigms for {BCI} communication,'' \emph{IEEE
Transactions on Neural Systems and Rehabilitation Engineering}, vol.~28,
no.~12, pp. 2647--2659, 2020.
\bibitem{alcaraz2022diffusion}
J.~M.~L. Alcaraz and N.~Strodthoff, ``Diffusion-based time series imputation
and forecasting with structured state space models,'' \emph{arXiv preprint
arXiv:2208.09399}, 2022.
\bibitem{jeong2021diff}
M.~Jeong, H.~Kim, S.~J. Cheon, B.~J. Choi, and N.~S. Kim, ``Diff-tts: A
denoising diffusion model for text-to-speech,'' \emph{arXiv preprint
arXiv:2104.01409}, 2021.
\bibitem{ronneberger2015u}
O.~Ronneberger, P.~Fischer, and T.~Brox, ``U-net: Convolutional networks for
biomedical image segmentation,'' in \emph{Medical Image Computing and
Computer-Assisted Intervention 2015: 18th International Conference, Munich,
Germany, October 5-9, 2015, Proceedings, Part III 18}.\hskip 1em plus 0.5em
minus 0.4em\relax Springer, 2015, pp. 234--241.
\bibitem{smith2017cyclical}
L.~N. Smith, ``Cyclical learning rates for training neural networks,''
\emph{2017 IEEE Winter Conference on Applications of Computer Vision}, pp.
464--472, 2017.
\bibitem{leeMH2019eeg}
M.-H. Lee, O.-Y. Kwon, Y.-J. Kim, H.-K. Kim, Y.-E. Lee, J.~Williamson,
S.~Fazli, and S.-W. Lee, ``{EEG} dataset and {OpenBMI} toolbox for three
{BCI} paradigms: an investigation into {BCI} illiteracy,''
\emph{GigaScience}, vol.~8, no.~5, p. giz002, 2019.
\bibitem{lee2021decoding}
D.-Y. Lee, M.~Lee, and S.-W. Lee, ``Decoding imagined speech based on deep
metric learning for intuitive {BCI} communication,'' \emph{IEEE Transactions
on Neural Systems and Rehabilitation Engineering}, vol.~29, pp. 1363--1374,
2021.
\bibitem{krepki2007berlin}
R.~Krepki, B.~Blankertz, G.~Curio, and K.-R. M{\"u}ller, ``The berlin
brain-computer interface ({BBCI})--towards a new communication channel for
online control in gaming applications,'' \emph{Multimedia Tools and
Applications}, vol.~33, no.~1, pp. 73--90, 2007.
\bibitem{Delorme2004EEGLAB}
A.~Delorme and S.~Makeig, ``{EEGLAB}: an open source toolbox for analysis of
single-trial {EEG} dynamics including independent component analysis,''
\emph{Journal of Neuroscience Methods}, vol. 134, no.~1, pp. 9--21, 2004.
\bibitem{coretto2017open}
G.~A.~P. Coretto, I.~E. Gareis, and H.~L. Rufiner, ``Open access database of
eeg signals recorded during imagined speech,'' \emph{12th International
Symposium on Medical Information Processing and Analysis}, vol. 10160, p.
1016002, 2017.
\end{thebibliography}