Thirty-two healthy, right-handed adults with normal sleeping patterns from Tianjin Artificial Intelligence Innovation Center (TAIIC) participated in this study. These participants included 21 males and 11 females between the ages of 21 and 42 years (Mean $\pm$ SD = 27.4 $\pm$ 4.2 years). All participants learned the experimental process and signed informed consent agreements prior to enrollment. Experimental approval was obtained from the Ethics Committee of Tianjin University (TJUE-2021-138).

A flow chart outlining experimental procedures is shown in Fig. 1. Seven members of our research group constructed an initial database of 129 movie clips. According to emotional movie database (EMDB) requirements [17], 29 of these videos were selected as alternative stimulating materials. Each video has been carefully reviewed to ensure the stability of content, consistent character portrayal, clear image and sound quality, and a fixed refresh rate [2]. Additionally, we have taken great care to ensure that each video clip does not contain both positive and negative emotional content. Twenty-seven annotators (age: 26.26 $\pm$ 3.30 years; 19 males, 8 females) assessed these 29 video clips by self-rating their emotions (valence, arousal) according to a self-assessment questionnaire, following the practice of decoding affective physiological responses (DECAF) [18]. Three annotators were regarded as outliers based on abnormal rating values. The consistency of the valence/arousal ratings among the remaining 24 annotators were 0.69 and 0.30 according to Krippendorff’s alpha metric [19] and suggested that the selected videos consistently induced designated emotions among annotators. More details regarding the emotion stimulating materials are outlined in previous work [20]. Twenty videos (sad $\times$ 5, fear $\times$ 5, happy $\times$ 5, neutral $\times$ 5) were further selected as stimulating materials according to their valence and arousal values in this study, with each video lasting for 150 seconds.

Fig. 1.

Flow chart of the experimental procedures. DCA, discriminant correlation analysis; PCA, principal component analysis; SVM, support vector machine.

Subjects wore a NEUROSCAN wireless electrode cap (Neuroscan SynAmps2 Model 8050, Compumedics USA, Inc., Charlotte, NC, USA) with 64 channels to collect EEG signals. The electrode cap followed the international 10–20 system, and the sampling rate was 1000 Hz. Four electrode channels (M1, M2, CB1, CB2) were excluded. Subjects watched video stimulating materials on a 15-inch screen with a refresh rate of 60 Hz. The ground electrode was located between Fz and FPz, and the reference electrode was located between Cz and CPz (Fig. 2A). Raw EEG data were preprocessed using MATLAB’s EEGLAB toolbox (https://sccn.ucsd.edu/eeglab) [21] with bandpass filtering between 0.3–50 Hz, 50 Hz trap filtering to remove powerline interference, down-sampling to 200 Hz to reduce computational complexity, and independent component analysis (ICA) to remove eye-movement artifacts. We only analyzed the EEG during the video viewing phase, and EEG signals labeled in the videos were considered ground truth.

Fig. 2.

The visualization of the extracted three types of EEG features. (A) Diagram of sixty-channel EEG cap. (B) Five frequency band topographic maps of group-averaged PSD values (left) and summing of PSDs across all channels (right). (C) The top 5% PLV connectivities and significant differences in the PLV feature of each band. (D) Eight microstates were extracted, and (E) Transition probability between microstates (Mean $\pm$ SEM). EEG, electroencephalography; PSD, power spectral density; PLV, phase-locking value; SEM, standard error of mean.

Discriminant Correlation Analysis (DCA) [16] was used to fuse multi-domain EEG features. The concept of DCA maximizes pairwise correlations between feature sets in the same class while eliminating correlation between feature sets in different classes. This results in limiting the intra-class correlation. Principal component analysis (PCA) was used for dimensionality reduction prior to feature fusion and for high-dimensional features that required large computing requirements, which may lead to the dimensionality curse. Following established practices [24], pairwise feature sets were first fused using DCA, and all fused features were then summed. We carried out leave-one-subject-out cross validation with linear-support vector machine (SVM) as a classifier. The average classification accuracy was calculated as an index for the performance evaluation of our emotion recognition model.

We calculated the PSDs of 60 channels in the five frequency bands outlined above and obtained 5 $\times$ 60 = 300 PSD features. For the four emotion types, the group-average PSD from individual EEG signals was calculated. Topographic PSD maps of five bands are shown in Fig. 2B. By adding up the PSD of all channels; the PSD tends to increase with increases in emotional valence (negative emotions neutral positive emotions) in all frequency bands.

The PLV was calculated in the form of a 60 $\times$ 60 matrix to represent functional connectivity for the classification of emotion. We extracted the upper triangular elements of five bands as features for classification, resulting in 5 $\times$ 60 $\times$ (60–1)/2 = 8850 dimensional feature vectors. Then we selected the top 5% strongest connectivity links for analysis (Fig. 2C). Highly correlated connections in the low-frequency bands $\delta$ and $\theta$ were principally distributed within the occipital lobe (electrodes of O, PO, and P), and showed strong similarities between different emotions. In contrast, highly correlated connections in the high-frequency bands $\alpha$, $\beta$ and $\gamma$ primarily distributed within the prefrontal and occipital lobes and showed more differentiation between emotions.

We conducted a one-way analysis of variance (ANOVA) to test whether there were significant differences in the connectivity strength among different emotions at each frequency band (Fig. 2C, right column). We set p 0.01 as the threshold for statistical significance. As outlined above, few connectivity differences were observed among emotions in the $\delta$ and $\theta$ bands. As many connections pass the significance test in $\alpha$, $\beta$ and $\gamma$ bands, we selected the top 5% significant connections, as calculated by F-statistic, for visualization to facilitate observation. Unlike group-averaged, highly correlated connections that are mainly distributed between neighboring electrodes (Fig. 2C, left three columns), significant connectivity differences among emotions are primarily distributed between distant electrodes (Fig. 2C, right column). Consistent with a previous study [23], these findings showed that significant differences were mainly observed in the connections between nodes located in the temporal lobe suggesting that the temporal lobe region contained more information regarding emotion recognition. In regards to the $\alpha$ band, in addition to connections related to the temporal lobe, regions within the prefrontal and occipital lobe showed more active interactions in emotional brain activities.

We obtained eight microstates, designated microstate 1 (MS1) through MS8, which represent all topographic maps (Fig. 2D). MS1 occupied approximately 20% of occurrence and others occupied approximately 11% each. MS1 and MS2 were symmetric in the occipital-to-prefrontal orientation, and MS8 was symmetric with regards to the parietal-to-peripheral axis, while other microstates were lateralized. The microstates of four emotions were analyzed, and we observed variety in MS5 through MS8, while consistency was observed across emotions in MS1 through MS4.

We also computed the transition probability between microstates, and the results are shown in Fig. 2E. The transition probability of neutral was lower than the other three emotions in the first column (MS2~MS8 $\to$ MS1). We also observed an increasing trend in the 5th column as well as a decreasing trend in the 8th column with the increase of emotional valence.

As shown in Fig. 3A, we measured an emotion classification accuracy of 44.30 $\pm$ 14.5% (Mean $\pm$ SD), 56.88 $\pm$ 13.79%, and 56.95 $\pm$ 14.12% for PSD, PLV and microstate, respectively. Two possibilities may account for the higher emotion classification accuracies of PLV and microstate compared with PSD. First, spatial and temporal features of EEG may be superior to spectral features in measuring emotion recognition. Second, the interaction of multiple brain regions may better represent changes in emotions rather than the activation of local brain regions. When we conducted pairwise feature fusion, PLV + microstate performed the best with an accuracy of 62.73 $\pm$ 13.45%. Although PSD reduced classification performance when used pairwise with either PLV or microstate, PSD still provided complementary information that improved the classification accuracy of PSD + PLV + microstate to 64.69 $\pm$ 11.99%. Of note, classification accuracy fell to 57.50 $\pm$ 13.46% when we directly concatenated the PSD + PLV + microstate features without conducting DCA, indicating that DCA improved EEG-based emotion classification performance (t (31) = 3.03, p 0.01, paired-sample t-test).

Fig. 3.

The classification accuracies and confusion matrix. Classification accuracy (A) and confusion matrix (B, C, D, E) of fused features and single mode features. *, p 0.01, **, p 0.001, paired-sample t-test.

We next generated a group-average confusion matrix using the fused features (Fig. 3B) and the single mode feature of PSD (Fig. 3C), PLV (Fig. 3D), and microstate (Fig. 3E). Columns and rows of the confusion matrix represented classified and ground truth labels, respectively. The entries in the diagonal showed percentages of emotion classes that were classified correctly. We observed that negative emotions, such as sadness and fear, exhibited relatively low classification accuracies in contrast to the other two non-negative emotions. This led us to develop the view that negative emotions are more difficult to recognize [25]. In general, PLV features had advantages in recognizing happy, sad, and neutral emotions, whereas microstate features had higher accuracy in recognizing fearful emotional states. This finding indicates the complementarity of different feature types. A feature-fusion approach stably improves the performance of emotion classification with different degrees of improvement; for example sad was 1.26%, fear was 3.14%, neutral was 6.92%, and happy was 5.03%. Taking into consideration that the emotion classification in this study was both cross-subject and four-class and that two types of negative emotions were included, our proposed model satisfied classification performance compared with other state-of-the-art work [23, 26]. For example, Chen et al. [23] conducted frequency-domain fusion on EEG features and obtained two-class (negative vs. positive) cross-subject emotion recognition accuracies of 71.14 $\pm$ 6.97%, 61.48 $\pm$ 10.97%, and 66.62 $\pm$ 9.10% on SEED, BCI2020-A, and BCI2020-B datasets, respectively. Based on EEG signals, Cai et al. [26] obtained subject-specific accuracies between 58% to 62% using a four-class (joy, peace, anger, and depression) cross-subject emotion recognition.