Abstract:

Decoding human internal emotional states based on EEG(Electroencephalogram) and surrounding physiological signals is key in the field of emotional computing, but the performance of machine learning models using EEG or surrounding physiological signal modes may be limited. In this study, a multi-mode fusion strategy is proposed based on the single mode method. The differential entropy, statistical and complexity features are extracted from each EEG fragment, and these features are properly integrated with the surrounding physiological signals. Multiple modal features recorded in the DEAP(Database for Emotion Analysis using Physiological Signals) data set are incorporated in the proposed method. In terms of titer, the experimental accuracy of single EEG feature is 49.21%, the classification accuracy of two types of feature fusion is 56.39%, 55.24% and 56.98%, and the experimental accuracy of three types of mode fusion is 56.98%. In terms of arousal, the experimental accuracy of single EEG feature is 49.34%, the classification accuracy of two types of feature fusion is 54.53%, 54.53% and 59.39%, and the experimental accuracy of three types of feature fusion is 55.48%. The experimental results show that the classification accuracy of multi-modal features after the fusion of EEG features and peripheral physiological features is the highest, and the classification accuracy is improved by 7.77% and 10.05%, respectively, compared with the single EEG features.

Key words: affective computing, feature fusion, emotion recognition, information fusion, multimodal emotion fusion, EEG, emotion modality, machine learning