Facial Expression and Peripheral Physiology Fusion to Decode Individualized Affective Experience
Extracted source text ยท arXiv 1811.07392
% use the "wcp" class option for workshop and conference
% proceedings
%\documentclass[gray]{jmlr} % test grayscale version
%\documentclass[tablecaption=bottom]{jmlr}% journal article
\documentclass[pmlr]{jmlr}% new name PMLR (Proceedings of Machine Learning)
%\documentclass{IEEEtran}
% The following packages will be automatically loaded:
% amsmath, amssymb, natbib, graphicx, url, algorithm2e
%\usepackage{rotating}% for sideways figures and tables
%\usepackage{longtable}% for long tables
% The booktabs package is used by this sample document
% (it provides \toprule, \midrule and \bottomrule).
% Remove the next line if you don't require it.
\usepackage{booktabs}
% The siunitx package is used by this sample document
% to align numbers in a column by their decimal point.
% Remove the next line if you don't require it.
\usepackage[load-configurations=version-1]{siunitx} % newer version
%\usepackage{siunitx}
% The following command is just for this sample document:
\newcommand{\cs}[1]{\texttt{\char`\\#1}}% remove this in your real article
% Define an unnumbered theorem just for this sample document for
% illustrative purposes:
\theorembodyfont{\upshape}
\theoremheaderfont{\scshape}
\theorempostheader{:}
\theoremsep{\newline}
\newtheorem*{note}{Note}
% \citepp
% change the arguments, as appropriate, in the following:
\jmlrvolume{86}
\jmlryear{2018}
\jmlrsubmitted{submission date}
\jmlrpublished{publication date}
\jmlrworkshop{2nd IJCAI Workshop on Artificial Intelligence in Affective Computing} % W&CP title
% The optional argument of \title is used in the header
\title[Individualized Affective Experience Decoding]{Facial Expression and Peripheral Physiology Fusion\\ to Decode Individualized Affective Experience}
\author{\Name{Yu Yin, Mohsen Nabian, Sarah Ostadabbas\nametag{\thanks{This paper has code available at GitHub under \href{https://github.com/ostadabbas/3d-facial-landmark-detection-and-tracking}{3D Facial Landmark Detection and Tracking}, provided by the corresponding author's lab.}}} \Email{yin.yu1@husky.neu.edu, monabiyan@ece.neu.edu, ostadabbas@ece.neu.edu}\\
\Name{Miolin Fan, ChunAn Chou} \Email{fan.mi@husky.neu.edu, ch.chou@northeastern.edu}\\
\Name{Maria Gendron} \Email{maria.gendron@gmail.com}\\
\addr Northeastern University, Boston, USA}
\usepackage{times}
\usepackage{xcolor}
\usepackage{soul}
\usepackage[utf8]{inputenc}
\usepackage[small]{caption}
%\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
% \usepackage{commath}
\newcommand{\eqnref}[1]{~(\ref{eqn:#1})}
\newcommand{\figref}[1]{Fig.~\ref{fig:#1}}
\newcommand{\tblref}[1]{Table~\ref{tbl:#1}}
\newcommand{\secref}[1]{Section~\ref{sec:#1}}
\newcommand{\algref}[1]{Algorithm.~\ref{alg:#1}}
\newcommand{\thmref}[1]{Theorem~\ref{thm:#1}}
\newcommand{\defref}[1]{Definition~\ref{definition:#1}}
\newcommand{\lemref}[1]{Lemma~\ref{lem:#1}}
%\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\com}[1]{\textcolor{red}{#1}}
\setcounter{page}{9}
\begin{document}
\input{figs}
\input{tabs}
\maketitle
\begin{abstract}
Affective experience prediction using different data modalities measured from an individual such as their facial expression or physiological signals has received substantial research attention in recent years. However, most studies ignore the fact that people besides having different responses under affective stimuli, may also have different resting dynamics (embedded in both facial and physiological patterns) to begin with. In this paper, we present a multimodal approach to simultaneously analyze facial movements and several peripheral physiological signals to decode individualized affective experiences under positive and negative emotional contexts, while considering their personalized resting dynamics. We propose a person-specific recurrence network to quantify the dynamics present in the person's facial movements and physiological data. Facial movement is represented using a robust head vs. 3D face landmark localization and tracking approach, and physiological data are processed by extracting known attributes related to the underlying affective experience. The dynamical coupling between different input modalities is then assessed through the extraction of several complex recurrent network metrics. Inference models are then trained using these metrics as features to predict individual's affective experience in a given context, after their resting dynamics are excluded from their response. We validated our approach using a multimodal dataset consists of (i) facial videos and (ii) several peripheral physiological signals, synchronously recorded from 12 participants while watching 4 emotion-eliciting video-based stimuli. The affective experience prediction results signified that our multimodal fusion method improves the prediction accuracy up to 19\% when compared to the prediction using only one or a subset of the input modalities. Furthermore, we gained prediction improvement for affective experience by considering the effect of individualized resting dynamics.
\end{abstract}
\begin{keywords}
Multimodal data fusion, individualized affective experience, facial expression, physiological signals.
\end{keywords}
\section{Introduction}
Affective experience is an important construct in explaining several critical aspects of human behaviors and is impaired or irregular in a number of neurodevelopmental and psychiatric disorders \citep{panksepp2004affective}. Experimental evidence indicates that positive or negative affective experience plays an important role in motivating future actions \citep{Barrett2009} and can promote behavioral patterns linked to compromised mental health \citep{Wichers2015}. In addition, information about affective states of users has become more and more important in human-computer interaction and many other emerging areas \citep{Schaaff2009} in recent years as it greatly facilitates the ability of computers to heed the rules of human communication \citep{Picard2000}.
%One contributor to affective experience is the self perception of the physiological changes in the periphery, a mechanism known as interoception \citep{Critchley2004, Kleckner2017}. However, we still do not fully understand the mechanisms supporting affective dynamics \citep{Kuppens2010}, including how physiological changes in the periphery and face contribute. Measuring and predicting affective states through the combination of facial expressions and peripheral physiological signals has received substantial research attention in the past few decades.
%Affect display is the external show of emotion or affect, whether voluntary or not. It is a concept used in psychology to describe the experience of emotion or feeling displayed to others through facial expressions, hand gestures, and physiological signals. An increased understanding of the mechanisms of affect displays is of critical clinical and public health relevance.
A common approach to quantify the range of human affective states and to predict human affective experiences under different circumstances is based on decoding their facial movements. Movements of the face are considered as a particularly rich source for affective display and are commonly referred to as ``facial expressions'' \citep{Littlejohn2008, Ekman2002,facialSynchroney,duran2017coherence,russell2017science,fernandez2013emotion}. However, not all emotions occur with an expression in face or even distinguishable facial expression \citep{Ekman1993} and the correspondence between specific facial expressions and underlying emotional experiences is not robust in psychology \citep{Reisenzein2013}. To address this problem, various physiological signals including electrocardiogram (ECG), electroencephalogram (EEG), electrodermal activity (EDA), blood pressure, and respiration patterns have been used as complementary information to decode affective states \citep{Verma2014,khalaf2017analysis} based on the function of multiple physiological systems in the body \citep{Liao2006,perez2017decoding}.
%Though different studies have focused on integrating information from multiple modalities to enhance the performance of automatic emotion recognition, the fact that large individualized differences in expression and physiology under similar affective experiences might also influence the recognition results has been played down, especially within affective computing society. In fact, human facial movements or physiological responses even without any emotional stimulus (i.e. under resting state) differs substantially from one individual to another, due to multiple intrinsic and extrinsic factors (e.g. gender, personality, culture influence, level of education, etc.) \citep{Hurlburt2015, Krys2016, Joo2012}. However, in the majority of the affective computing or even psychological studies, individual differences are only accounted by subtracting the mean value (and very rarely considering the standard deviation) of the collected data during resting state from the stimulus-responding state's data, assuming that the resting dynamics can simply be modeled solely with the zeroth and first moments.
Among humans, there are considerable individualized differences in facial expression and peripheral physiological responses under similar affective experiences, which influence the recognition results of generalized predictive algorithms. Therefore, There is a large body of literature investigating personalized models and their impact on accurately predicting person-specific affect \citep{Yang2014, Hernandez2011, Kandemir2014}. However, it is crucial to note that human facial movements or physiological responses even without any emotional stimulus (i.e. under resting state) differ substantially from one individual to another, due to multiple intrinsic and extrinsic factors (e.g. gender, personality, cultural influences, level of education, etc.) \citep{Hurlburt2015, Krys2016, Joo2012}. In the majority of the affective computing or even psychological studies, individual baseline differences are only accounted by subtracting the mean value (and very rarely considering the standard deviation) of the collected data during resting state from the stimulus-responding state data, assuming that the resting dynamics can simply be modeled solely with the zeroth and first moments.
In this paper, we applied recurrence network analysis for multimodal data (i.e. facial movements and physiological signals) fusion to identify and decode individualized affective experiences. To extract features from facial movements, we developed a robust landmark tracking approach, in which head movement is also independently tracked and decoupled from the facial landmark movements. As for the physiological signals including ECG, EDA, and respiration, a series of signal-specific algorithms developed in \citep{NabianMohsen2017Abpt, bioSp} were utilized to extract psychologically-related features. We used complex network metrics to assess the inter-system dynamical coupling between different response
modalities of a person under a negative or positive affective experience. We used these metrics to build an inference model for individual affective experience prediction. Critically, we also modeled the resting dynamics of each individual participant before undergoing any emotional induction, to account for individualized baseline differences in affective experience. Our main contributions in this paper are as follows: (1) employing a 3D model for facial landmark localization/tracking, which decouples head motion from face expression; (2) assessing the resting/affective multimodal response of each individual through a higher order dynamics using recurrence network metrics; and (3) developing a novel multimodal feature fusion approach based on recurrence network for affective experience decoding.
\section{Related Work}
Emotion recognition studies based on facial expression analysis often take a categorical approach, where a label from a set of six purported basic emotions (anger, disgust, fear, happiness, sadness, surprise) is assigned to a pattern of facial movements \citep{russell1994there}. Yet in real life, emotions are much more complex \citep{Barrett2016}, and specificity and consistency of facial movements to emotions is often lacking \citep{fernandez2013emotion, Reisenzein2013}. Moreover, some of the emotions do not even fit well in any of the basic categories \citep{Koelstra2013}.
A finer-grained assessment of facial expressions is to directly detect specific facial muscle actions (action units; AUs), including but not limited to the facial movements on which the basic emotion expressions were based \citep{tian2001recognizing,Cohn2007,sanchez2016cascaded}. The facial features used in AUs recognition studies are often either geometric features indicating the location of facial characteristic points (mouth, eyes, chin, etc.) or appearance features representing the facial textures \citep{Zeng2009, Valstar2015}. In \citep{Pantic2006}, a set of facial points was used as geometric feature to recognize AUs in frontal-view face images. In \citep{Bartlett2006, Guo2005}, appearance-based methods, such as Gabor wavelets or eigenfaces were applied to classify facial expressions or AUs. Some other work used both geometric and appearance features. In \citep{Ringeval2015}, appearance features were extracted using local Gabor binary patterns and geometric features were extracted based on 49 facial landmarks. Similarly, \citep{Benitez-Quiroz2016} used second-order statistics of facial landmarks (i.e., distances and angles between landmark points) for geometric features and Gabor filters for appearance features.
However, most 2D feature-based methods are only suitable for the analysis of frontal-view face, which means only a small range of head movement is allowed. Fewer works of facial expression analysis have been done based on 3D face models. In \citep{Zeng2006}, a 3D face tracker was used to handle the arbitrary behavior of the person in the natural setting. Both geometry and appearance features were extracted based on the 3D face model. In \citep{Cohn2004}, authors focused on recognizing two of the most important facial actions in brow (brow raising and brow lowering) measured in spontaneous facial behavior with non-frontal pose, moderate out-of-plane head motion, and occlusion. A cylindrical head model was applied to estimate head movements in \citep{Xiao2002}.
%\citep{Gu2006}The authors present an approach for aligning a 3D deformable model to a single face image. The model consists of a set of sparse 3D points and the view-based patches associated with every point.
%To date, only a few studies attempted the fusion of facial and physiological signals in order to improve emotion experience recognition performance.
To date, research on fusion of facial expression and physiological data in order to improve performance of emotion recognition algorithms is continuing to attract the attention of academia and industry alike.
In \citep{Koelstra2013}, multimodal approaches based on both feature-level and decision-level fusion were applied to analyze facial expressions and EEG signals for generation of affective tags. In feature-level fusion, the authors simply stacked all the feature vectors together. In decision-level fusion, they first classified each modality individually and then combined the classifier outputs in a linear fashion. In \citep{Liao2006}, authors focused on recognizing only two different affective states, stress and fatigue. They applied a decision-level fusion approach to recognize these two affective states from multiple modalities including physical appearance (e.g. facial expression, head movement), physiological measures (e.g. EEG, ECG), behavioral data (e.g. mouse movement, type speed), and user performance (e.g. response time). In \citep{Miao2018}, authors proposed to apply recurrence network analysis to quantify the dynamics and to extract nonlinear features from EEG signals for classifying affective states. Based on existing knowledge and methods, the fusion of multimodal person-specific data in levels prior to the emotion experience inference is largely unexplored \citep{Zeng2006}. It is worth mentioning that data fusion in earlier stages, before decision, can lead to capturing the higher order information presents in the multimodal data, as well as the dynamical coupling between different input modalities.
%In \citep{Verma2014}, Multimodal fusion of physiological signals is proposed.
\section{Materials \& Methods}
\rating
\subsection{Dataset}
In this work, we used a multimodal dataset collected by the Psychology Department of Northeastern University, which contains both facial video recording and synchronous physiological data including ECG, EDA, and respiration signals. These data were obtained from 12 consenting participants during two data collection phases: (Phase I) each participant described their two most positive and their two most negative emotional experiences, and (Phase II) each participant watched their own 4 recorded videos as stimuli.
In both phased, facial videos were recorded by a frontal camera at 25 frames per second. The three physiological signals were sampled at 1000Hz using BioLab v.3.0.13 (Mindware Technologies; Gahanna, OH) via a BioNex 8-Slot chassis (Model50-3711-08).
The recorded videos in Phase I (that formed our video-based stimuli) were played back to the same individual who recorded them (i.e., participants watched themselves), such that each participant viewed video-based stimuli in Phase II. Participants provided continuous ratings of their affective feelings while watching their video-based stimuli, using a rating dial (ranging from unpleasant to neutral to pleasant). Continuous ratings were obtained since this method is non-disruptive (are not requiring stopping of the stimulus), allows online ratings that previous research suggests are less subject to recall bias, and can provide idiographic data at a high temporal resolution \citep{ruef2007continuous}. These self-ratings, as shown in \figref{rating}, reveal dynamics across the video-based stimulus segments in the degree of positive or negative affect.
% indicating that they will serve as a robust source stimulus material.
\subsection{Facial Landmark Localization and Tracking}
In order to track relative movements of the facial landmarks solely generated by the facial muscles in the videos, we developed a robust tracking approach, in which the head movement is also tracked and decoupled from the facial landmark movements. We first employed a state-of-the-art 2D facial alignment algorithm presented in \citep{kazemi2014one} to automatically localize 68 landmarks for each frame of the face video. Then, a 3D face model based on \citep{kittler20163d} is used to estimate the depth information from the 2D frames and thus to achieve 3D landmark tracking.
\landmarkFramework
\subsubsection{2D Facial Landmark Localization}
A cascade of trained regressors is utilized to localize the facial landmarks for each video frame as described in \citep{kazemi2014one} (see \figref{landmarkFramework}). To train each regressor, the gradient tree boosting algorithm is used with a sum of square error loss \citep{john2010elements}. Assume we have training dataset \{($I_1,S_1$), ... , ($I_n,S_n$)\}, where each $I_i$ is a face image and $S_i$ is its shape vector. We set an initial shape estimate $\widehat{S}_{i}^{(0)}$ for every face image. In each regression tree, the regression function $r_t$ is learned using the gradient tree boosting algorithm, and then the estimation of every shape is updated as:
\begin{equation}
\widehat{S}_{i}^{(t+1)} = \widehat{S}_{i}^{(t)} + r_t(I_i,\widehat{S}_{i}^{(t)})
\end{equation}
The initial shape $\widehat{S}_{i}^{(0)}$ for each frame is simply chosen as the mean shape of the training dataset centered and scaled according to the bounding box of the full face, detected with the histogram of oriented gradients (HOG) features. In each level of cascade, estimated landmarks are refined by adding residuals produced by the previous regression tree. Note that all frames in video are normalized to have the same Euclidean distance of pixels between the middle of the two eyes. Hence the landmark movements would be comparable across a given individual. \figref{MohsenLandmark} shows the result of 68 landmark detection on a video frame captured by a laptop webcam.
\MohsenLandmark
%\vspace{-.2in}
\subsubsection{3D Facial Landmark Tracking}
Our facial landmarks tracking algorithm needs to remain invariant across head movement including its translation, scaling (getting closer or further from camera), and rotations (i.e. roll, yaw, pitch). To eliminate the interference of head movement, we first extract the depth information of each face pixels from 2D video frames using the 3D morphable face model described in \citep{kittler20163d}. This model consists of a principal component analysis (PCA) model of face shapes, which could be used for reconstructing a 3D face from a single 2D image. The PCA model consists of a set of principal components $V = [\upsilon_1,...,\upsilon_K]$, the mean value of all the facial meshes $\overline{\upsilon}$, and their standard deviation $\sigma_k$. The shape of a novel face is then generated with:
\begin{equation}
S_i=\overline{\upsilon} +\sum_{k=1}^{K} \alpha_k \sigma_k \upsilon_k,
\end{equation}
where $K$ is the number of principal components and $\alpha_k$'s are the representation of $S_i$ in the coordinates of the PCA shape space. The 3D face shapes were then reconstructed by fitting 68 detected landmarks to a PCA shape model. For the purpose of model fitting, the gold standard algorithm of Hartley and Zisserman \citep{hartley2003multiple} were implemented, which finds a least squares approximation of an affine camera matrix given 2D-3D point pairs. An example result of the landmark fitting is shown in \figref{YuModel}.
\YuModel
Using the 3D geometric transformation matrix, every frame could be transformed into frontal face, enabling us to track and compare the movement of the facial landmarks throughout the video. The landmarks detection algorithm is not accurate for some frames with large head rotation angles or poor lighting. We address this issue by applying landmark geometric constraints.
% might be wrongly detected due to or . The detection of some landmarks on that frame will be on an unreasonable position. We will apply to address this issue.
\subsubsection{Facial Expression Features}
% Human emotions and intentions are more often communicated by changes in one or a few discrete facial features.
Rather than few prototypic facial expressions, such as happiness, anger, surprise, and fear, it is shown that the dynamics and temporal combination of facial action units (AUs) may provide more reliable and specific quantification of the expressive movements of the human face during emotion \citep{tian2001recognizing}.
% Facial AUs can be formulated based on the facial landmarks. For instance, width of the mouth as an AU is correlated with the horizontal distance of the lips corners. Therefore,
Guided by the work done by Y. Tian \citep{tian2001recognizing}, we reduced the facial landmarks feature dimensions from 2$\times$68 to 12 features described as follows: (1-2) left and right eyebrow y-values (corresponding to AUs 1, 2 and 5); (3) inner corners differences of eyebrows (corresponding to AU 4); (4) horizontal distance of the the two corners of lips (corresponding to AUs 12 and 20); (5) vertical distance of the two lips (corresponding to AUs 25, 26 and 27); (6) average vertical positions of the two corners of the lips (corresponding to AU 15); (7-9) head rigid displacement in X, Y, and Z direction, respectively; (10-12) head rigid rotation in roll, pitch, and yaw direction, respectively.
% Facial action units are the result of facial muscle activation and facial muscle movements can be detected by combining relative or aggregate displacements of groups of facial landmark \citep{tian2001recognizing}.
It is noted that combining the facial landmarks into AUs leads to the reduction of the stochastically distributed noise in landmark positioning as well as resolving the issue of subject-specific and camera-specific variations.
%Moreover, comparing to other open source tools that detect AUs as discrete values, our methods provides continues measures of facial features corresponding to AUs.
Moreover, comparing to other open source tools that can directly detect AUs, our method can provide more information related to the facial movements for two reasons. Firstly, most AU detection algorithms could only deal with frontal-view face, while our 3D landmark tracking method could also extract head movement directly from the video. Secondly, our method provides continues measures of facial landmarks instead of only 6 discrete numbers representing AUs and its intensities.
% muscle movements, grouping of the landmarks may provide the following advantageous :
% As shown in figure ~\ref{lndNo}, we obtain the 3D position of the 68 face landmarks for each frame of the videos.
% However, to study the emotions, the relative displacements of landmarks has to be studied.
% Moreover, the face action units arise from a combined movements of face muscles and consequently, groups of landmarks will move together accordingly.
% \lndNo
% 1) Adding or subtracting the landmarks positions may suppress the noises.(ensemble averaging)
% 2) Features obtained from the relative distances between landmarks are independent from the individuals face specifications.
% As a result the following 12 features extracted from the face landmarks and head movements are as follows:
% \begin{equation}
% featuresLDMK=[max(y_l_ebrew_n_1),max(y_r_ebrew_n_1),max(x_difference_eyebrew_corners_n_1)];
% featuresLDMK=[featuresLDMK,max(mouthWidth_1),max(mouthY_1),max(mouthOpen_1)];
% featuresHead=[max(teta_x_1),max(teta_y_1),max(teta_z_1)];
% featuresHead=[featuresHead,max(delta_x_1),max(delta_y_1),max(delta_z_1)];
% \end{equation}
% After obtaining the 3D fa
% A total of 68 features were extracted for landmarks. Firstly, I computed the Euclidean movement of each landmark. And set the maximum movement of each landmark as features. As for head movement, the maximum movements of head position and rotation angles (6) were set as head features.
\subsection{Peripheral Physiological Signal Processing}
\subsubsection{Signal Preprocessing}
To eliminate or reduce the noise and artifacts carried by the physiological signal measurements, signal-specific filtering is required prior to applying any feature extraction algorithm. An elliptic bandpass filter with the cut-off frequencies of 5Hz and 45Hz was applied on the ECG signals. This cut-off frequency range was selected based on the power spectral density analysis of the ECG signals and the elliptic filter type was selected to ensure the amplitude of the peak points on the signal were not significantly suppressed by the filter \citep{Chavan2006elliptic}. With the similar investigation, a low-pass filter with cutoff frequency of 1Hz was selected as the optimal choice and was applied on the EDA signals \citep{deluca2010emg}. As for the respiration signal, we applied a Butterworth lowpass filter with a cutoff frequency of 20Hz.
% In the pre-processing of the ECG signals, an Elliptic band-pass filter with cut-off frequencies of 5-45Hz was used. These cut-off frequency values were selected based on the power spectral density (PSD) analysis of the ECG signals and the Elliptic filter was selected to ensure the amplitude of the peak points on the signal were not significantly suppressed by the filter \citep{Chavan2006elliptic}.
% An elliptic bandpass filter with cutoff frequencies of 10 -- 300Hz was used for the EDA signals.
\subsubsection{Psychologically-Inspired Physiological Features}
The dimensionality reduction of the peripheral physiological time series signals is done by employing a series of signal-specific algorithms to extract informative physiological features from each signal \citep{NabianMohsen2017Abpt}. Physiological signals including ECG, EDA, and respiration were processed using the biosignal processing MATLAB toolbox accessible at \citep{AClabtoolbox}.
ECG signals contain rich information relevant to human health, sleep quality, and emotional states \citep{Ra2013ECG}. For the ECG signals, the features are based upon the detected QRS points on the signal \citep{1985real} and after successfully detecting these points, relevant physiological features can be computed (see \tblref{BioFeatures}).
%These features include 1) mean R-R intervals (the time between consecutive heartbeats), 2) standard deviation of R-R intervals, 3) standard deviation of the differences between adjacent R-R intervals, 4) square root of the mean of the sum of the squares of differences between adjacent R-R intervals, 5) the number of pairs of adjacent R-R intervals where the first R-R interval exceeds the second R-R interval by more than 50ms, 6) the number of pairs of adjacent R-R intervals where the second R-R interval exceeds the first R-R interval by more than 50ms, 8) mean area of each QRS complex and its 9) standard deviation.
% \figECGwithQRS
% \figSCR
\tabBioFeatures
The EDA signal is composed of two types of activity, tonic and phasic. The slowly varying base signal is the tonic aspect, and is also called skin conductance level (SCL). The faster-changing part is called phasic activity or skin conductance response (SCR). SCRs are related to more acute exterior stimuli or non-specific activation \citep{gamboa2005electrodermal}. Many important features for this purpose are extracted from SCRs. The occurrence of the SCR is detected by finding two consecutive zero-crossings, from negative to positive and positive to negative of the bartletted differentiated EDA signal. Most of EDA features listed in \tblref{BioFeatures} are based on the detection of SCRs.
It is noted that for given windows of EDA in which no SCR was found, feature values were set to 0. %NaN and we further replaced them with the same value as the EDA features in their previous window.
Features were also extracted from respiration signal as its pattern may vary in distinct affective states. Regular respiration is linked to relaxation, while fast and shallow breathing might correspond to more aroused emotions, such as acute anxiety and emotional tension \citep{Koelstra2013}. Important features extracted from respiration signal are provided in \tblref{BioFeatures}.
\subsubsection{Windowing and Overlapping Sizes for Feature Extraction}
Multimodal data fusion using a recurrent network requires all feature vectors to be the same length. Therefore, values for window sizing and overlapping were carefully selected for different facial expression signals as well as physiological signals to achieve same size feature vectors in all of the modalities as well as to reasonably capture the temporal dynamics of the signals. Please note that the larger window size is chosen for the signal with slower changing rate to make sure appropriate features are correctly extracted. To align different modalities in time, we applied different overlaps to force corresponding windows of different modalities to have the same starting point in time. Windowing and overlapping sizes for each signal modality are provided in \tblref{FeatureExtraction}.
\tabFeatureExtraction
\graphFlow
\subsection{Multimodal Data Fusion using Recurrent Network}
Here, we demonstrate a method introduced earlier by \citep{Zou2015} to build an inter-system network model that represents the joint contribution between different response modalities of a person under an affective experience context. In our study, the nodes in the network represent features extracted from multimodal data (facial expressions and physiological responses) and edges are defined based on the directional coupling between modalities. The network construction consists of the following steps: (1) time-delayed embedding for reconstructing phase space trajectory \citep{Takens1981}; (2) recurrence plot (RP) construction \citep{Eckmann1987}; (3) extension of RP to multiple systems (i.e. modalities) to obtain a joint RP (JRP) \citep{Romano_2004}; and (4) extraction of complex network metrics to assess the inter-system dynamical coupling. The general framework of the proposed fusion approach applied on our multimodal dataset is shown in \figref{graphFlow}.
% (4) computation of a mean conditional probability of recurrence (MCR) \citep{Zou2015};
\subsubsection{Recurrence Plot for Single Modality}
Introduced by Eckmann in 1987 \citep{Eckmann1987}, RP is a visualization to represent the temporal dependency relationships between all states in a time series data using a binary, squared matrix \citep{Eckmann1987}. Suppose the state of system (or modality) $X$ at time \textit{i} and {j} is represented by $\textbf{x}_i, \textbf{x}_j$, recurrence can be recorded by the binary function as:
\begin{equation}
\textbf{R}_{i,j}^{X} = \Theta (\epsilon_X -{||\textbf{x}_i - \textbf{x}_j||_1}), \textbf{x}_i \in \mathbb{R}^m, \textit{i}, \textit{j} = 1,...N,
\end{equation}
where $\Theta$ is a Heaviside function and the RP puts a point at coordinates $(i,j)$ if ${R}_{i,j}^{X} = 1$, any time the state trajectory gets sufficiently close (within the system threshold $\epsilon_X$) to a point it has been previously.
\subsubsection{Extension to Multimodality and Investigation of Coupling}
Although the original method was developed for a single time series, later variations of RP included consideration of multivariate time series from different aspects. To best capture the dynamic coupling between multiple modalities, we adopted JRP since it represents when a recurrence occurs simultaneously in two or more time series \citep{Romano_2004}. Suppose we have modalities $X$ and $Y$, for which the individual RPs can be obtained. In general, JRP is obtained by the product of multiple systems:
\begin{equation}
\textbf{JR}_{i,j}^{X,Y} = \Theta (\epsilon_X -{||\textbf{x}_i - \textbf{x}_j||_1}) \Theta (\epsilon_Y -{||\textbf{y}_i - \textbf{y}_j||_1}).
\end{equation}
%In the present study, each input modality is treated as a dynamic system such that an inter-system network was constructed by pair-wise production of RPs of each modality. The unidirectional coupling strength in between a pair of nodes (i.e. modalities $X$ and $Y$) in JPR is captured by the mean conditional probability of recurrence (MCR) metric \citep{Zou2015}:
%\begin{equation}
%\textbf{M}_{CR}(Y|X) = \dfrac{1}{N} \sum_{i=1}^{N}
%\dfrac{\sum_{j=1}^{N}\textbf{JR}_{i,j}^{X,Y}}{\sum_{j=1}^{N}\textbf{R}_{i,j}^{X}},
%\end{equation}
%and the difference in MCR for two-way directions on an edge represents the coupling direction (bidirectional coupling is allowed here). $\textbf{M}_{CR}(Y|X) < \textbf{M}_{CR}(X|Y)$ means the modality \textit{X} drives the modality \textit{Y}, and vise versa.
\subsubsection{Complex Network-based Feature Extraction}
We enabled the usage of complex network analysis by converting the JRP matrix to the adjacency matrix for a network, which serves as a graphical representation of the temporal neighborhood relations between system states across entire time series. This network, referred to as recurrence networks (RN), is expressed in the following formula:
\begin{equation}
\textbf{A}_{i, j}(\varepsilon) = \textbf{JR}_{i, j}(\varepsilon) - \textbf{I},
\end{equation}
where $\textbf{A}_{i, j} (\varepsilon)$ is the adjacency matrix, $\textbf{JR}_{i, j} (\varepsilon)$ is the JRP (a binary matrix), and $\textbf{I}(T)$ is an identity matrix for removing the elements on the main diagonal line that creates self-loops in the network.
In our study, RN is used to describe the dynamical behaviors (or patterns) of a person under an affective experience context. Further, to characterize the network features, a set of metrics are computed for quantitative assessment of the network topology, and further provide information about coupling dynamics in a different view. The network measures we computed include two general classes: (1) global measures: transitivity, global efficiency, and out-strength/in-strength correlation, and (2) local measures: in-strength/out-strength, local efficiency, edge/node betweenness centrality, diversity, and clustering coefficients. The global measures are related to the topological structures of the entire network, while the local measures are related to the attribute of individual nodes. These metrics are widely used for describing the connectivity patterns in complex systems; the computational details are included in \citep{rubinov2010complex}.
\subsection{Inference Models for Affective Experience Prediction}
%We first built a baseline network for each individual, which assumes that the baseline can be different for everyone. Comparing to the baseline network of oneself, the change in network features (e.g. the within- or between modality connectivity patterns, measured by degree, density, etc.) will be used for task prediction across subjects.
% We hope to find some common patterns for the change in positive/negative video rating tasks. Moreover, individual differences are expected and to be discussed in the result \& discussion section (considering the limited sample size, the statistical significance may not be achieved).
\subsubsection{Prediction based on Emotional Video Stimulus Content}
We performed a binary across-individual classification on video stimuli with positive or negative contents. Since each subject experienced two positive and two negative video stimuli, the class distribution was balanced.
First, we extracted the signal-specific features from the multimodal input signals collected from each subject while he/she was watching a video stimulus. To fuse these features, we then constructed their JRPs, as well as the network metrics containing global and local measures. We used a support vector machine (SVM) with a linear kernel function as the inference model and the stimuli context prediction accuracy was defined as the percentage of stimuli correctly classified based on their positive or negative contents. We performed a per-subject leave-one-out cross validation method for the model evaluation.
\subsubsection{Prediction based on Affective Self-Rating Scales}
Here, we trained an inference model to predict individualized affective experiences of participants while watching a video stimulus, rather than classifying the content of the stimulus. This inference model accounts for individual differences in responding to emotional stimuli, by assuming that the multimodal facial and physiological data collected during an affective experience is more directly linked to the person's internal affective state rather than the content of the stimulus as positive or negative.
The same signal-specific features are fused by JRP for predicting the valence self-rating of the person. The original range of valence rating was [0, 50]. We applied a Min-Max normalization to scale the valence ratings to [0, 1] for each subject. Since the actual rating range of different subjects could differ a lot, normalized values allow the comparison across subjects in a way that eliminates the effects of certain gross influences. For each video stimulus, participants provided continuous ratings of their positive/negative affect. We trained our inference model to predict the median value of continuous ratings, since self-ratings might become dramatically high or low at the end of the video. Instead of predicting the mean value of continuous ratings, we assumed the median value is a more robust representation of the subject's valenced experience during the whole video stimulus.
We employed support vector regressor (SVR) with a ridge penalty as the self-rating regression model. The same leave-one-out cross-validation approach is applied as in the previous classification task. Root mean square error (RMSE) and mean absolute error (MAE) of the predicted valence self-rating scores are employed here to indicate the performance of our trained regression model.
\section{Experimental Results}
\tabclassificationResult
\subsection{Emotional Video Stimulus Content Prediction Results}
We performed a per-subject leave-one-out cross validation, where the classifier is trained on a total of 44 trials from 11 participants, and tested on the 4 videos from the remaining one participant. For each participant, classification accuracy and F1 score were used to evaluate the performance of our stimulus content classification model. \tblref{classificationResult} gives the classification results over each modality and video stimulus content. As a baseline, we also gave the expected values of the random classifier. According to the significance test with the resulted p-values in \tblref{classificationResult}, the obtained classification accuracies of the trained model are not significantly more accurate than the random classifier.
% For some modalities such as EDA, respiration, or facial landmarks, the classification results are even less than 50\%.
Overall, the result shows that video content ratings might not be predictable from the recorded data. One explanation for the low prediction accuracy of our trained model could be the fact that the data was not recorded from the participants while they were verbally explaining their positive or negative experiences, but rather we used the facial and physiological signal recordings of the participants while they were watching back their own recorded videos. In other words, all the recorded signals are not directly related to the video-based stimulus contents. In the following section, we instead predict the self-ratings, which may be more closely related to the recorded multimodal signals.
% Noted that the class distribution is balanced in the dataset, voting according to the majority class and voting with the ratio of the classes would have similar result with random classifier.
%not necessary
\tabregressionResult
\subsection{Affective Self-Rating Scale Prediction Results}
\tblref{regressionResult} provides the results for our regression model to predict self-ratings of the subjective experience of emotion trained on the multimodal fusion data. We reported the RMSE and MAE, as the two widely used metrics for the accuracy evaluation of continuous variable estimation. Both of these values are negatively-oriented scores, in which the lower values represent higher accuracy performance. As a baseline, we also gave the expected values of a random regressor, which were found by randomly generating the value that are drawn from a normal distribution of the training data.
As evident in the \tblref{regressionResult}, the prediction accuracy measures obtained from the models trained on either single modality and multimodal fusion data are higher than the prediction accuracy of the random classifier, except for the models trained solely on head movement features. Moreover, the highest accuracy was achieved with our proposed network-based multimodal fusion method. Note that the physiological signals are also well ranked in terms of performance for rating prediction: respiration performs second best on valence and EDA performs third best. This implies that alongside the facial video data, the physiological signals provide informative complementary descriptions of the affective experience.
In addition to our multimodal fusion method based on the recurrence network, another basic fusion method was also employed and evaluated. In the basic method, each modality of facial (i.e. landmarks and head movement) or physiological signals (i.e. ECG, EDA and Respiration) is weighed equally for the fusion. The results of facial and physiological features fused by the basic fusion method were also given in \tblref{regressionResult}. It is observed that the prediction accuracy from fusion of facial signals outperforms the prediction accuracy based on single modalities. However, the model based on fusion of the physiological signals only outperformed the model trained on the ECG features and had lower performance than models trained on EDA and respiration data.
\section{Conclusion}
In this paper, we presented a multimodal approach that analyzes both facial expressions and peripheral physiological signals concurrently to identify and decode the individualized affective experiences of participants watching a series of emotional video-based stimuli. We developed a robust 3D face tracking approach, in which head movement is also independently tracked and decoupled from the facial landmark movements. Signal-specific features were then extracted from both facial and physiological signals (ECG, EDA, and respiration). We applied recurrence network for multimodal data fusion and complex network-based features were extracted from the fusion of different modalities. Finally, we validated our approach using a multimodal dataset consists of (i) facial videos and (ii) several peripheral physiological signals, synchronously recorded from 12 participants while watching 4 emotion-eliciting video stimuli. The experimental results for binary classification of video-based stimuli with positive vs. negative content showed that video content ratings might not be predictable from the recorded data from the participants when they are watching them.% as the facial and physiological signals were not recorded when the participants talked about their positive or negative experiences. %One potential explanation is that people might not feel so much when they watching themselves talking about past experiences.
One potential explanation is that people might recall their feelings while watching themselves talking about past experiences, but they may not feel the same at that moment. However, the recorded signal modalities were shown to be reliable predictors of the self-rating of their affective experience at the moment of watching the videos. Our feature-level fusion approach based on the recurrence network demonstrated to improve upon single modality results, suggesting the these modalities contain complementary information in accounting for person's affective experiences.
\bibliographystyle{named}
\small
\bibliography{paper}
\end{document}
\newcommand{\rating}{
\begin{figure}[t]
\centering
\includegraphics[width=0.55\linewidth,trim={0in 0in 0in 0in}, clip=true]{figures/rating.pdf}
\caption{Continuous rating of an example video stimulus using rating dial (ranging from negative = 0 to positive = 50) reveals dynamics in videos across time.
}
\vspace{-.2in}
\label{fig:rating}
\end{figure}
}
\newcommand{\landmarkFramework}{
\begin{figure}[t]
\centering
\includegraphics[width=0.9\linewidth,trim={0in 0in 0in 0in}, clip=true]{figures/landmarkFramework.png}
\caption{The framework of the cascade of regression trees designed for 2D facial landmark detection. In each level of cascade, estimated landmarks are refined by adding residuals produced by the previous regression tree.}
\vspace{-.2in}
\label{fig:landmarkFramework}
\end{figure}
}
\newcommand{\MohsenLandmark}{
\begin{figure}[h]
\centering
\includegraphics[width=0.45\linewidth,trim={0in 0in 0in 0in}, clip=true]{figures/Mohsen.pdf}
\caption{Results of 2D landmark detection on a webcam video frame. It outputs 68 landmarks inside a bounding box detected using HOG features.}
\label{fig:MohsenLandmark}
\end{figure}
}
\newcommand{\YuModel}{
\begin{figure}[t]
\centering
\begin{tabular}{@{}cc@{}}
\multicolumn{1}{c}{\includegraphics[width=0.28\linewidth]{figures/Yu_3D_1.pdf}}
\includegraphics[width=0.4\linewidth,trim={0in 0in 0in 0in}, clip=true]{figures/Yu_3D_2.pdf}&
\includegraphics[width=0.28\linewidth,trim={0in 0in 0in 0in}, clip=true]{figures/Yu_3D_3.pdf}
\end{tabular}
\caption{An example result of the landmark fitting: resulting shape and model fitting (left), 3D model fitting result (middle), and frontal view of the 3D facial model (right).}
\vspace{-.2in}
\label{fig:YuModel}
\end{figure}
}
\newcommand{\graphFlow}{
\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth,trim={0in 2.5in 0in 1.5in}, clip=true]{figures/graphFlow.pdf}
\caption{The framework for multimodal data fusion using recurrent network. First, signal-specific facial and physiological features are windowed and extracted. Second, the phase space trajectory for each modality is reconstructed and the corresponding recurrence plot is obtained. Then, a joint recurrence plot (JRP) is calculated by multiplying recurrence plots together. Finally, complex network metrics are extracted to assess the inter-system dynamical coupling.}
\vspace{-.2in}
\label{fig:graphFlow}
\end{figure*}
}
\newcommand{\lndNo}{
\begin{figure}[ht]
\centering
\includegraphics[width=1\linewidth,trim={0in 2in 0in 1in}, clip=true]{figures/lndNo.jpg}
\caption{The framework for feature fusion.}
\vspace{-.2in}
\label{fig:lndNo}
\end{figure}
}
\newcommand{\figSCR}{
\begin{figure}
\centering
\includegraphics[width=\linewidth,trim=0in 0in 0in 0in, clip=true]{figures/SCR_detection.pdf}
\caption{Detection of SCRs on bartletted differentiated EDA signal (top) and demonstration of detected SCRs' peaks and starts on low-passed EDA signal (bottom).}
\label{fig:SCR}
\vspace{-.2in}
\end{figure}
}
\newcommand{\figECGwithQRS}{
\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth,trim=0in 0in 0in 0in, clip=true]{figures/ECGwithQSR.pdf}
\caption{Raw ECG signal (top); bandpass filtered ECG signal (bottom) with the characteristic points (P,Q,R,S and T) detected on the signal.}
\label{fig:ECGwithQRS}
\vspace{-.2in}
\end{figure}
}
\newcommand{\tabFeatureExtraction}{
\begin{table}[h]
\caption{Signal-specific feature extraction parameters.}
\vspace{-.2in}
\begin{center}
\begin{tabular}{c | c c c }
& Window Size & Overlap & Features \\
Modality& (sec) & (\%) & (\#) \\
\hline
\textbf{Face} & 5 & 50 & 6 \\
\textbf{Head} & 5 & 50 & 6 \\
\textbf{ECG} & 5 & 50 & 10 \\
\textbf{EDA} & 20 & 88.2 & 5 \\
\textbf{Resp} & 30 & 92.4 & 4
\end{tabular}
\label{tbl:FeatureExtraction}
\vspace{-.2in}
\end{center}
\end{table}
}
\newcommand{\tabBioFeatures}{
\begin{table}[ht]
\caption{Physiology-specific feature extraction from physiological signal modalities.}
\vspace{-.2in}
\begin{center}
\begin{tabular}{ p{1.4cm} p{12cm}}
\hline
\hline
Modality & Extracted Features\\
\hline
\textbf{ECG} & Features based on QRS detection: mean R-R intervals (the time between consecutive heartbeats), standard deviation of R-R intervals, standard deviation of the differences between adjacent R-R intervals, the square root of the mean of the sum of the squares of differences between adjacent R-R intervals, the number of pairs of adjacent R-R intervals where the first R-R interval exceeds the second R-R interval by more than 50ms, the number of pairs of adjacent R-R intervals where the second R-R interval exceeds the first R-R interval by more than 50ms, mean area of each QRS complex and its standard deviation. \\
\hline
\textbf{EDA} & Signal mean, numbers of detected SCRs, mean SCR duration, mean SCR amplitude, mean SCR rise-time (where rise-time of an SCR is defined as the time between the initial rise and the peak of an SCR). \\
\hline
\textbf{Resp} & Respiration rate (peak to peak in ms), amplitude (height of peak), percent inhalation (the proportion of rising part of the signal in each cycle) and percent exhalation (the proportion of falling part of the signal in each cycle).\\
\hline
\hline
\end{tabular}
\label{tbl:BioFeatures}
\vspace{-.2in}
\end{center}
\end{table}
}
\newcommand{\tabclassificationResult}{
\begin{table}[b]
\caption{Emotional video-based stimulus content prediction results.}
\vspace{-.2in}
\begin{center}
\begin{tabular}{c | c c | c c c}
Modality & Accuracy & p-value & F1 & Precision & Recall \\
\hline
\textbf{ECG} & 51.4 &0.29& 51.3 & 51.3 & 52.1\\
\textbf{EDA} & 48.6 &0.18& 47.7 & 48.6 & 47.2\\
\textbf{Resp} & 49.3 &0.20& 43.4 & 49.1 & 40.3\\
\textbf{Face} & 45.1 &0.10& 44.6 & 44.9 & 44.4\\
\textbf{Head} & 50.7 &0.27& 52.4 & 50.6 & \textbf{55.6}\\
\hline
\textbf{Fusion} & \textbf{55.3}&0.49 & \textbf{54.1} & \textbf{56.7} & 51.8\\
\textbf{Random} & 50.4 &0.24& 50 & 50.6 & 48.5
\end{tabular}
\label{tbl:classificationResult}
\vspace{-.2in}
\end{center}
\end{table}
}
\newcommand{\tabregressionResult}{
\begin{table}[t]
\caption{Affective self-rating scale prediction results.}
\vspace{-.2in}
\begin{center}
\begin{tabular}{c | c c }
Modality & RMSE & MAE \\
\hline
\textbf{ECG} & 0.33 & 0.28 \\
\textbf{EDA} & 0.27 & 0.24 \\
\textbf{Resp} & 0.27 & 0.24 \\
\textbf{Face} & 0.32 & 0.27 \\
\textbf{Head} & 0.45 & 0.33 \\
\hline
\textbf{Facial} & 0.27 & \textbf{0.24} \\
\textbf{Physio} & 0.29 & 0.25 \\
\textbf{Fusion} & \textbf{0.26} & \textbf{0.24} \\
\textbf{Random} & 0.40 & 0.37
\end{tabular}
\label{tbl:regressionResult}
\vspace{-.2in}
\end{center}
\end{table}
}
\begin{thebibliography}{58}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[ACl()]{AClabtoolbox}
{Biosignal-Specific Processing (Bio-SP) Tool}.
\newblock \url{http://www.northeastern.edu/ostadabbas/software/}.
\newblock Accessed: 2018.
\bibitem[Barrett and Bliss-Moreau(2009)]{Barrett2009}
Lisa~Feldman Barrett and Eliza Bliss-Moreau.
\newblock {Affect as a Psychological Primitive}.
\newblock \emph{Advances in Experimental Social Psychology}, 41:\penalty0
167--218, 2009.
\newblock ISSN 00652601.
\newblock \doi{10.1016/S0065-2601(08)00404-8}.
\bibitem[Barrett et~al.(2016)Barrett, Quigley, and Hamilton]{Barrett2016}
Lisa~Feldman Barrett, Karen~S. Quigley, and Paul Hamilton.
\newblock {An active inference theory of allostasis and interoception in
depression}.
\newblock \emph{Philosophical Transactions of the Royal Society B: Biological
Sciences}, 371\penalty0 (1708):\penalty0 20160011, 2016.
\bibitem[Bartlett et~al.(2006)Bartlett, Littlewort, Frank, Lainscsek, Fasel,
and Movellan]{Bartlett2006}
Marian~Stewart Bartlett, Gwen Littlewort, Mark Frank, Claudia Lainscsek, Ian
Fasel, and Javier Movellan.
\newblock {Fully Automatic Facial Action Recognition in Spontaneous Behavior}.
\newblock \emph{7th International Conference on Automatic Face and Gesture
Recognition (FGR06)}, pages 223--230, 2006.
\newblock \doi{10.1109/FGR.2006.55}.
\bibitem[Benitez-Quiroz et~al.(2016)Benitez-Quiroz, Srinivasan, and
Martinez]{Benitez-Quiroz2016}
C.~Fabian Benitez-Quiroz, Ramprakash Srinivasan, and Aleix~M. Martinez.
\newblock {EmotioNet: An Accurate, Real-Time Algorithm for the Automatic
Annotation of a Million Facial Expressions in the Wild}.
\newblock \emph{2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR)}, pages 5562--5570, 2016.
\newblock ISSN 10636919.
\newblock \doi{10.1109/CVPR.2016.600}.
\bibitem[Chavan et~al.(2005)Chavan, Agarwala, and Uplane]{Chavan2006elliptic}
Mahesh~S. Chavan, R.~A. Agarwala, and M.~D. Uplane.
\newblock Digital elliptic filter application for noise reduction in ecg
signal.
\newblock \emph{4th WSEAS International Conference on ELECTRONICS, CONTROL and
SIGNAL PROCESSING}, pages 58--63, 2005.
\bibitem[Cohn(2007)]{Cohn2007}
Jeffrey~F Cohn.
\newblock {Foundations of human-centered computing: Facial expression and
emotion.}
\newblock \emph{International Joint Conference on Artificial Intelligence},
pages 233--238, 2007.
\newblock \doi{10.1145/1180995.1181043}.
\bibitem[Cohn et~al.(2004)Cohn, Reed, Ambadar, Xiao, and Moriyama]{Cohn2004}
Jeffrey~F Cohn, Lawrence~Ian Reed, Zara Ambadar, Jing Xiao, and Tsuyoshi
Moriyama.
\newblock {Automatic analysis and recognition of brow actions and head motion
in spontaneous facial behavior}.
\newblock \emph{2004 IEEE International Conference on Systems, Man and
Cybernetics (IEEE Cat. No.04CH37583)}, 1:\penalty0 0--6, 2004.
\newblock ISSN 1062-922X.
\newblock \doi{10.1109/ICSMC.2004.1398367}.
\bibitem[De~Luca et~al.(2010)De~Luca, Gilmore, Kuznetsov, and
Roy]{deluca2010emg}
Carlo~J De~Luca, L~Donald Gilmore, Mikhail Kuznetsov, and Serge~H Roy.
\newblock Filtering the surface emg signal: Movement artifact and baseline
noise contamination.
\newblock \emph{Journal of Biomechanics}, 43\penalty0 (8):\penalty0 1573--1579,
2010.
\bibitem[DUR{\'A}N et~al.(2017)DUR{\'A}N, Reisenzein, and
FERN{\'A}NDEZ-DOLS]{duran2017coherence}
JUANI DUR{\'A}N, Rainer Reisenzein, and JOS{\'E}MIGUEL FERN{\'A}NDEZ-DOLS.
\newblock Coherence between emotions and facial expressions.
\newblock \emph{The Science of Facial Expression}, page 1849, 2017.
\bibitem[Eckmann et~al.(1987)Eckmann, Kamphorst, and Ruelle]{Eckmann1987}
J-P Eckmann, S~Oliffson Kamphorst, and David Ruelle.
\newblock Recurrence plots of dynamical systems.
\newblock \emph{EPL (Europhysics Letters)}, 4\penalty0 (9):\penalty0 973, 1987.
\bibitem[Ekman(1993)]{Ekman1993}
Paul Ekman.
\newblock {Facial expression and emotion.}
\newblock \emph{The American psychologist}, 48\penalty0 (4):\penalty0 384--392,
1993.
\newblock ISSN 0003-066X.
\newblock \doi{10.1037/0003-066X.48.4.384}.
\bibitem[Ekman et~al.(2002)Ekman, Friesen, and Hager]{Ekman2002}
Paul Ekman, Wallace~V. Friesen, and Joseph~C. Hager.
\newblock \emph{{Facial Action Coding System - Investigator's Guide}}.
\newblock 2002.
\newblock ISBN 0-931835-01-1.
\newblock \doi{10.1016/j.msea.2004.04.064}.
\bibitem[Fan and Chou(2018)]{Miao2018}
Miaolin Fan and Chun-An Chou.
\newblock {Recognizing Affective State Patterns Using Regularized Learning with
Nonlinear Dynamical Features of EEG}.
\newblock \emph{IEEE Conference on Biomedical and Health Informatics (BHI)
2018}, 2018.
\bibitem[Fern{\'a}ndez-Dols and Crivelli(2013)]{fernandez2013emotion}
Jos{\'e}-Miguel Fern{\'a}ndez-Dols and Carlos Crivelli.
\newblock Emotion and expression: Naturalistic studies.
\newblock \emph{Emotion Review}, 5\penalty0 (1):\penalty0 24--29, 2013.
\bibitem[Gamboa and Fred(2005)]{gamboa2005electrodermal}
HFS Gamboa and ALN Fred.
\newblock An electrodermal activity psychophysiologic model.
\newblock 2005.
\bibitem[Guo and Dyer(2005)]{Guo2005}
Guodong Guo and Charles~R. Dyer.
\newblock {Learning from examples in the small sample case: Face expression
recognition}.
\newblock \emph{IEEE Transactions on Systems, Man, and Cybernetics, Part B:
Cybernetics}, 35\penalty0 (3):\penalty0 477--488, 2005.
\newblock ISSN 10834419.
\newblock \doi{10.1109/TSMCB.2005.846658}.
\bibitem[Hartley and Zisserman(2003)]{hartley2003multiple}
Richard Hartley and Andrew Zisserman.
\newblock \emph{Multiple view geometry in computer vision}.
\newblock Cambridge university press, 2003.
\bibitem[Hernandez et~al.(2011)Hernandez, Morris, and Picard]{Hernandez2011}
Javier Hernandez, Rob~R. Morris, and Rosalind~W. Picard.
\newblock {Call center stress recognition with person-specific models}.
\newblock \emph{Lecture Notes in Computer Science (including subseries Lecture
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)}, 6974
LNCS\penalty0 (PART 1):\penalty0 125--134, 2011.
\newblock ISSN 03029743.
\newblock \doi{10.1007/978-3-642-24600-5_16}.
\bibitem[Hurlburt et~al.(2015)Hurlburt, Alderson-Day, Fernyhough, and
K{\"{u}}hn]{Hurlburt2015}
Russell~T. Hurlburt, Ben Alderson-Day, Charles Fernyhough, and Simone
K{\"{u}}hn.
\newblock {What goes on in the resting-state? A qualitative glimpse into
resting-state experience in the scanner}.
\newblock \emph{Frontiers in Psychology}, 6\penalty0 (OCT), 2015.
\newblock ISSN 16641078.
\newblock \doi{10.3389/fpsyg.2015.01535}.
\bibitem[John~Lu(2010)]{john2010elements}
ZQ~John~Lu.
\newblock The elements of statistical learning: data mining, inference, and
prediction.
\newblock \emph{Journal of the Royal Statistical Society: Series A (Statistics
in Society)}, 173\penalty0 (3):\penalty0 693--694, 2010.
\bibitem[Joo et~al.(2012)Joo, Yeon, and Lee]{Joo2012}
Hong~Jin Joo, Bora Yeon, and Kyoung~Uk Lee.
\newblock {The impact of personality traits on emotional responses to
interpersonal stress}.
\newblock \emph{Clinical Psychopharmacology and Neuroscience}, 10\penalty0
(1):\penalty0 54--58, 2012.
\newblock ISSN 20934327.
\newblock \doi{10.9758/cpn.2012.10.1.54}.
\bibitem[Kandemir et~al.(2014)Kandemir, Vetek, G{\"{o}}nen, Klami, and
Kaski]{Kandemir2014}
Melih Kandemir, Akos Vetek, Mehmet G{\"{o}}nen, Arto Klami, and Samuel Kaski.
\newblock {Multi-task and multi-view learning of user state}.
\newblock \emph{Neurocomputing}, 139:\penalty0 97--106, 2014.
\newblock ISSN 18728286.
\newblock \doi{10.1016/j.neucom.2014.02.057}.
\bibitem[Kazemi and Sullivan(2014)]{kazemi2014one}
Vahid Kazemi and Josephine Sullivan.
\newblock One millisecond face alignment with an ensemble of regression trees.
\newblock \emph{Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 1867--1874, 2014.
\bibitem[Khalaf et~al.(2017)Khalaf, Nabian, Fan, Yin, Wormwood, Siegel,
Quigley, Barrett, Akcakaya, Chou, and Ostadabbas]{khalaf2017analysis}
Aya Khalaf, Mohsen Nabian, Miaolin Fan, Yu~Yin, Jolie Wormwood, Erika Siegel,
Karen Quigley, Lisa~Feldman Barrett, Murat Akcakaya, Chun-An Chou, and Sarah
Ostadabbas.
\newblock Analysis of multimodal physiological signals within and across
individuals to predict psychological threat vs. challenge.
\newblock \emph{Available via PsyArXiv}, 2017.
\bibitem[Kittler et~al.(2016)Kittler, Huber, Feng, Hu, and
Christmas]{kittler20163d}
Josef Kittler, Patrik Huber, Zhen-Hua Feng, Guosheng Hu, and William Christmas.
\newblock 3d morphable face models and their applications.
\newblock \emph{International Conference on Articulated Motion and Deformable
Objects}, pages 185--206, 2016.
\bibitem[Koelstra and Patras(2013)]{Koelstra2013}
Sander Koelstra and Ioannis Patras.
\newblock {Fusion of facial expressions and EEG for implicit affective
tagging}.
\newblock \emph{Image and Vision Computing}, 31\penalty0 (2):\penalty0
164--174, 2013.
\newblock ISSN 02628856.
\newblock \doi{10.1016/j.imavis.2012.10.002}.
\bibitem[Krys et~al.(2016)Krys, Vauclair, Capaldi, Lun, Bond,
Dom{\'\i}nguez-Espinosa, Torres, Lipp, Manickam, Xing, et~al.]{Krys2016}
Kuba Krys, C-Melanie Vauclair, Colin~A Capaldi, Vivian Miu-Chi Lun,
Michael~Harris Bond, Alejandra Dom{\'\i}nguez-Espinosa, Claudio Torres,
Ottmar~V Lipp, L~Sam~S Manickam, Cai Xing, et~al.
\newblock Be careful where you smile: Culture shapes judgments of intelligence
and honesty of smiling individuals.
\newblock \emph{Journal of nonverbal behavior}, 40\penalty0 (2):\penalty0
101--116, 2016.
\bibitem[Liao et~al.(2006)Liao, Zhang, Zhu, Ji, and Gray]{Liao2006}
Wenhui Liao, Weihong Zhang, Zhiwei Zhu, Qiang Ji, and Wayne~D. Gray.
\newblock {Toward a decision-theoretic framework for affect recognition and
user assistance}.
\newblock \emph{International Journal of Human Computer Studies}, 64\penalty0
(9):\penalty0 847--873, 2006.
\newblock ISSN 10715819.
\newblock \doi{10.1016/j.ijhcs.2006.04.001}.
\bibitem[Littlejohn and Foss(2010)]{Littlejohn2008}
Stephen~W Littlejohn and Karen~A Foss.
\newblock \emph{Theories of human communication}.
\newblock Waveland press, 2010.
\bibitem[Nabian et~al.(2017)Nabian, Nouhi, Yin, and
Ostadabbas]{NabianMohsen2017Abpt}
Mohsen Nabian, Athena Nouhi, Yu~Yin, and Sarah Ostadabbas.
\newblock A biosignal-specific processing tool for machine learning and pattern
recognition.
\newblock \emph{Healthcare Innovations and Point of Care Technologies
(HI-POCT), 2017 IEEE}, pages 76--80, 2017.
\bibitem[Nabian et~al.(2018)Nabian, Yin, Wormwood, Quigley, Barrett, and
Ostadabbas]{bioSp}
Mohsen Nabian, Yu~Yin, Jolie Wormwood, Karen Quigley, Lisa Barrett, and Sarah
Ostadabbas.
\newblock An open-source feature extraction tool for the analysis of peripheral
physiological data.
\newblock \emph{IEEE Journal of Translational Engineering in Health and
Medicine}, 2018.
\bibitem[Nielson et~al.(2018)Nielson, Nabian, Yin, Wormwood, DeSteno, Barrett,
Quigley, and Ostadabbas]{facialSynchroney}
Catie Nielson, Mohsen Nabian, Yu~Yin, Jolie Wormwood, D~DeSteno, Lisa Barrett,
Karen Quigley, and Sarah Ostadabbas.
\newblock Extracting facial synchrony from videos of naturalistic dyadic
interaction.
\newblock \emph{2018 annual conference of the Society for Affective Science
(SAS)}, 2018.
\bibitem[Pan and Tompkins(1985)]{1985real}
Jiapu Pan and Willis~J Tompkins.
\newblock A real-time qrs detection algorithm.
\newblock \emph{Biomedical Engineering, IEEE Transactions on}, \penalty0
(3):\penalty0 230--236, 1985.
\bibitem[Panksepp(2004)]{panksepp2004affective}
Jaak Panksepp.
\newblock \emph{Affective neuroscience: The foundations of human and animal
emotions}.
\newblock Oxford university press, 2004.
\bibitem[Pantic and Patras(2006)]{Pantic2006}
Maja Pantic and Ioannis Patras.
\newblock {Dynamics of facial expression: Recognition of facial actions and
their temporal segments from face profile image sequences}.
\newblock \emph{IEEE Transactions on Systems, Man, and Cybernetics, Part B:
Cybernetics}, 36\penalty0 (2):\penalty0 433--449, 2006.
\newblock ISSN 10834419.
\newblock \doi{10.1109/TSMCB.2005.859075}.
\bibitem[Perez-Rosero et~al.(2017)Perez-Rosero, Rezaei, Akcakaya, and
Ostadabbas]{perez2017decoding}
Maria~S Perez-Rosero, Behnaz Rezaei, Murat Akcakaya, and Sarah Ostadabbas.
\newblock Decoding emotional experiences through physiological signal
processing.
\newblock \emph{Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE
International Conference on}, pages 881--885, 2017.
\bibitem[Picard(2000)]{Picard2000}
Rosalind~W Picard.
\newblock {Toward computers that recognize and respond to user emotion}.
\newblock \emph{IBM Systems Journal}, 39\penalty0 (3.4):\penalty0 705--719,
2000.
\newblock ISSN 0018-8670.
\newblock \doi{10.1147/sj.393.0705}.
\bibitem[Rameshwari S~Mane(2013)]{Ra2013ECG}
Vaibhav D Awandekar Priya~Rani Rameshwari S~Mane, A N~Cheeran.
\newblock Cardiac arrhythmia detection by ecg feature extraction.
\newblock \emph{International Journal of Engineering Research and Applications
(IJERA)}, 3\penalty0 (2):\penalty0 327--332, 2013.
\bibitem[Reisenzein et~al.(2013)Reisenzein, Studtmann, and
Horstmann]{Reisenzein2013}
Rainer Reisenzein, Markus Studtmann, and Gernot Horstmann.
\newblock {Coherence between emotion and facial expression: Evidence from
laboratory experiments}.
\newblock \emph{Emotion Review}, 5\penalty0 (1):\penalty0 16--23, 2013.
\newblock ISSN 17540739.
\newblock \doi{10.1177/1754073912457228}.
\bibitem[Ringeval et~al.(2015)Ringeval, Schuller, Valstar, Jaiswal, Marchi,
Lalanne, Cowie, and Pantic]{Ringeval2015}
Fabien Ringeval, Bj{\"o}rn Schuller, Michel Valstar, Shashank Jaiswal, Erik
Marchi, Denis Lalanne, Roddy Cowie, and Maja Pantic.
\newblock {AV+EC 2015: The First Affect Recognition Challenge Bridging Across
Audio, Video, and Physiological Data}.
\newblock \emph{Proc. AVEC 2015, satellite workshop of ACM-Multimedia 2015},
pages 3--8, 2015.
\newblock \doi{10.1145/2808196.2811642}.
\bibitem[Romano et~al.(2004)Romano, Thiel, Kurths, and von Bloh]{Romano_2004}
M~Carmen Romano, Marco Thiel, J{\"u}rgen Kurths, and Werner von Bloh.
\newblock Multivariate recurrence plots.
\newblock \emph{Physics letters A}, 330\penalty0 (3-4):\penalty0 214--223,
2004.
\bibitem[Rubinov and Sporns(2010)]{rubinov2010complex}
Mikail Rubinov and Olaf Sporns.
\newblock Complex network measures of brain connectivity: uses and
interpretations.
\newblock \emph{Neuroimage}, 52\penalty0 (3):\penalty0 1059--1069, 2010.
\bibitem[Ruef and Levenson(2007)]{ruef2007continuous}
Anna~Marie Ruef and Robert~W Levenson.
\newblock Continuous measurement of emotion.
\newblock \emph{Handbook of emotion elicitation and assessment}, pages
286--297, 2007.
\bibitem[Russell(1994)]{russell1994there}
James~A Russell.
\newblock Is there universal recognition of emotion from facial expression? a
review of the cross-cultural studies.
\newblock \emph{Psychological bulletin}, 115\penalty0 (1):\penalty0 102, 1994.
\bibitem[Russell and Fern{\'a}ndez-Dols(2017)]{russell2017science}
James~A Russell and Jos{\'e}-Miguel Fern{\'a}ndez-Dols.
\newblock The science of facial expression.
\newblock \emph{Oxford University Press}, 2017.
\bibitem[S{\'a}nchez-Lozano et~al.(2016)S{\'a}nchez-Lozano, Martinez,
Tzimiropoulos, and Valstar]{sanchez2016cascaded}
Enrique S{\'a}nchez-Lozano, Brais Martinez, Georgios Tzimiropoulos, and Michel
Valstar.
\newblock Cascaded continuous regression for real-time incremental face
tracking.
\newblock \emph{European Conference on Computer Vision}, pages 645--661, 2016.
\bibitem[Schaaff and Schultz(2009)]{Schaaff2009}
Kristina Schaaff and Tanja Schultz.
\newblock {Towards emotion recognition from electroencephalographic signals}.
\newblock \emph{Proceedings - 2009 3rd International Conference on Affective
Computing and Intelligent Interaction and Workshops, ACII 2009}, 2009.
\newblock ISSN 2156-8103.
\newblock \doi{10.1109/ACII.2009.5349316}.
\bibitem[Takens(1981)]{Takens1981}
Floris Takens.
\newblock Detecting strange attractors in turbulence.
\newblock \emph{Dynamical systems and turbulence, Warwick 1980}, pages
366--381, 1981.
\bibitem[Tian et~al.(2001)Tian, Kanade, and Cohn]{tian2001recognizing}
Y~Tian, Takeo Kanade, and Jeffrey~F Cohn.
\newblock Recognizing action units for facial expression analysis.
\newblock \emph{IEEE Transactions on pattern analysis and machine
intelligence}, 23\penalty0 (2):\penalty0 97--115, 2001.
\bibitem[Valstar et~al.(2015)Valstar, Almaev, Girard, McKeown, Mehu, Yin,
Pantic, and Cohn]{Valstar2015}
Michel~F. Valstar, Timur Almaev, Jeffrey~M. Girard, Gary McKeown, Marc Mehu,
Lijun Yin, Maja Pantic, and Jeffrey~F. Cohn.
\newblock {FERA 2015 - second Facial Expression Recognition and Analysis
challenge}.
\newblock \emph{2015 11th IEEE International Conference and Workshops on
Automatic Face and Gesture Recognition (FG)}, pages 1--8, 2015.
\newblock \doi{10.1109/FG.2015.7284874}.
\bibitem[Verma and Tiwary(2014)]{Verma2014}
Gyanendra~K. Verma and Uma~Shanker Tiwary.
\newblock {Multimodal fusion framework: A multiresolution approach for emotion
classification and recognition from physiological signals}.
\newblock \emph{NeuroImage}, 102\penalty0 (P1):\penalty0 162--172, 2014.
\newblock ISSN 10959572.
\newblock \doi{10.1016/j.neuroimage.2013.11.007}.
\bibitem[Wichers et~al.(2015)Wichers, Kasanova, Bakker, Thiery, Derom, Jacobs,
and {Van Os}]{Wichers2015}
Marieke Wichers, Zuzana Kasanova, Jindra Bakker, Evert Thiery, Catherine Derom,
Nele Jacobs, and Jim {Van Os}.
\newblock {From affective experience to motivated action: Tracking
reward-seeking and punishment-avoidant behaviour in real-life}.
\newblock \emph{PLoS ONE}, 10\penalty0 (6), 2015.
\newblock ISSN 19326203.
\newblock \doi{10.1371/journal.pone.0129722}.
\bibitem[Xiao et~al.(2002)Xiao, Kanade, and Cohn]{Xiao2002}
Jing Xiao, Takeo Kanade, and Jeffrey~F. Cohn.
\newblock {Robust full-motion recovery of head by dynamic templates and
re-registration techniques}.
\newblock \emph{Proceedings - 5th IEEE International Conference on Automatic
Face Gesture Recognition, FGR 2002}, pages 163--169, 2002.
\newblock ISSN 08999457.
\newblock \doi{10.1109/AFGR.2002.1004149}.
\bibitem[Yang et~al.(2014)Yang, Rudovic, Pavlovic, and Pantic]{Yang2014}
Shuang Yang, Ognjen Rudovic, Vladimir Pavlovic, and Maja Pantic.
\newblock {Personalized Modeling of Facial Action Unit Intensity}.
\newblock \emph{Advances in Visual Computing}, pages 269--281, 2014.
\newblock ISSN 16113349.
\bibitem[Zeng et~al.(2006)Zeng, Fu, Roisman, Wen, Hu, and Huang]{Zeng2006}
Zhihong Zeng, Yun Fu, Glenn~I. Roisman, Zhen Wen, Yuxiao Hu, and Thomas~S.
Huang.
\newblock {Spontaneous emotional facial expression detection}.
\newblock \emph{Journal of Multimedia}, 1\penalty0 (5):\penalty0 1--8, 2006.
\newblock ISSN 17962048.
\newblock \doi{10.4304/jmm.1.5.1-8}.
\bibitem[Zeng et~al.(2009)Zeng, Pantic, Roisman, and Huang]{Zeng2009}
Zhihong Zeng, Maja Pantic, Glenn~I. Roisman, and Thomas~S. Huang.
\newblock {A survey of affect recognition methods: Audio, visual, and
spontaneous expressions}.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine
Intelligence}, 31\penalty0 (1):\penalty0 39--58, 2009.
\newblock ISSN 01628828.
\newblock \doi{10.1109/TPAMI.2008.52}.
\bibitem[Zou et~al.(2015)Zou, Romano, Thiel, and Kurths]{Zou2015}
Yong Zou, M~Carmen Romano, Marco Thiel, and J{\"u}rgen Kurths.
\newblock Identifying coupling directions by recurrences.
\newblock \emph{Recurrence Quantification Analysis}, pages 65--99, 2015.
\end{thebibliography}