融合语音信号和脑电信号的多模态情感识别

doi:10.19665/j.issn1001-2400.2019.01.023

Abstract

Abstract:

To construct an effective emotion recognition system, the emotions of joy, sadness, anger and neutrality are induced by sound stimulation, and the corresponding speech and EEG signals are collected. First, this paper extracts the nonlinear geometric feature and nonlinear attribute feature of EEG and speech signals by phase space reconstruction respectively, and the emotion recognition is realized by combining the basic features. Then, a feature fusion algorithm based on the Restricted Boltzmann Machine is constructed to realize multimodal emotion recognition from the perspective of feature fusion. Finally, a multimodal emotion recognition system is constructed through decision fusion by using the quadratic decision algorithm. The results show that the overall recognition rate of the multimodal emotion recognition system constructed by feature fusion is 1.08% and 2.75% higher than that of speech signals and that of EEG signals respectively, and that the overall recognition rate of the multimodal emotion recognition system constructed by decision fusion is 6.52% and 8.19% higher than that of speech signals and that of EEG signals respectively. The overall recognition effect of the multimodal emotion recognition system based on decision fusion is better than that of feature fusion. A more effective emotion recognition system can be constructed by combining the emotional data of different channels such as speech signals and EEG signals.

Key words: speech signals, electroencephalo-graph signals, feature fusion, decision fusion

CLC Number:

TP391.4

MA Jianghe,SUN Ying,ZHANG Xueying. Multimodal emotion recognition for the fusion of speech and EEG signals[J].Journal of Xidian University, 2019, 46(1): 143-150.

Figures/Tables 9

References 21

[1]	ZHANG S, ZHANG S, HUANG T , et al. Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching[J]. IEEE Transactions on Multimedia, 2018,20(6):1576-1590. doi: 10.1109/TMM.2017.2766843
[2]	黄程韦, 金赟, 王青云 , 等. 基于语音信号与心电信号的多模态情感识别[J]. 东南大学学报:自然科学版, 2010,40(5):895-900. doi: 10.3969/j.issn.1001-0505.2010.05.003
	HUANG Chengwei, JIN Yun, WANG Qingyun , et al. Multimodal Emotion Recognition Based on Speech and ECG Signals[J]. Journal of Southeast University: Natural Science Edition, 2010,40(5):895-900. doi: 10.3969/j.issn.1001-0505.2010.05.003
[3]	PEREZ-GASPAR L A, CABALLERO-MORALES S O, TRUJILLO-ROMERO F . Multimodal Emotion Recognition with Evolutionary Computation for Human-robot Interaction[J]. Expert Systems with Applications, 2016,66:42-61. doi: 10.1016/j.eswa.2016.08.047
[4]	HUANG X, KORTELAINEN J, ZHAO G , et al. Multi-modal Emotion Analysis from Facial Expressions and Electroencephalogram[J]. Computer Vision and Image Understanding, 2016,147:114-124. doi: 10.1016/j.cviu.2015.09.015
[5]	刘仁怡 . 基于脑电和语音信号的情感识别研究[D]. 天津: 天津师范大学, 2015.
[6]	姚慧, 孙颖, 张雪英 . 情感语音的非线性动力学特征[J]. 西安电子科技大学学报, 2016,43(5):167-172. doi: 10.3969/j.issn.1001-2400.2016.05.029
	YAO Hui, SUN Ying, ZHANG Xueying . Research on Nonlinear Dynamics Features of Emotional Speech[J]. Journal of Xidian University, 2016,43(5):167-172. doi: 10.3969/j.issn.1001-2400.2016.05.029
[7]	HATAMIKIA S, NASRABADI A M. Recognition of Emotional States Induced by Music Videos Based on Nonlinear Feature Extraction and SOM Classification [C]//Proceedings of the 2014 21st Iranian Conference on Biomedical Engineering. Piscataway: IEEE, 2014: 333-337.
[8]	孙颖, 宋春晓 . 相空间重构的情感语音特征提取及优化[J]. 西安电子科技大学学报, 2017,44(6):162-168. doi: 10.3969/j.issn.1001-2400.2017.06.028
	SUN Ying, SONG Chunxiao . Emotional Speech Feature Extraction and Optimization of Phase Space Reconstruction[J]. Journal of Xidian University, 2017,44(6):162-168. doi: 10.3969/j.issn.1001-2400.2017.06.028
[9]	TAKENS F. Detecting Strange Attractors in Turbulence [C]//Lecture Notesin Mathematics: 898. Heidelberg: Springer-Verlag, 1981: 366-381.
[10]	LAHMIRI S . Generalized Hurst Exponent Estimates Differentiate EEG Signals of Healthy and Epileptic Patients[J]. Physica A: Statistical Mechanics and Its Applications, 2018,490:378-385. doi: 10.1016/j.physa.2017.08.084
[11]	KORDA A I, ASVESTAS P A, MATSOPOULOS G K , et al. Automatic Identification of Eye Movements Using the Largest Lyapunov Exponent[J]. Biomedical Signal Processing and Control, 2018,41:10-20. doi: 10.1016/j.bspc.2017.11.004
[12]	WOLF A, SWIFT J B, SWINNEY H L , et al. Determining Lyapunov Exponents from a Time Series[J]. Physica D: Nonlinear Phenomena, 1985,16(3):285-317. doi: 10.1016/0167-2789(85)90011-9
[13]	DAVID G F, PAU M M, JORGE J N , et al. Noisy EEG Signals Classification Based on Entropymetrics. Performance Assessment Using First and Second Generation Statistics[J]. Computers in Biology and Medicine, 2017,87:141-151. doi: 10.1016/j.compbiomed.2017.05.028 pmid: 28595129
[14]	HARRINGTON P D B . Feature Expansion by a Continuous Restricted Boltzmann Machine for Near-infrared Spectrometric Calibration[J]. Analytica Chimica Acta, 2018,1010:20-28. doi: 10.1016/j.aca.2018.01.026
[15]	ZHENG H, ZHANG S, SUN X. Classification Recognition of Anchor Rod Based on PSO-SVM [C]//Proceedings of the 2017 29th Chinese Control and Decision Conference. Piscataway: IEEE, 2017: 2207-2212.
[16]	TENG K, WANG J. Classification Related Manifold Dimension Estimation with Restricted Boltzmann Machine [C]//Proceedings of the 2013 7th International Conference on Image and Graphics. Washington: IEEE Computer Society, 2013: 857-862.
[17]	CAI X, HU S, LIN X. Feature Extraction Using Restricted Boltzmann Machine for Stock Price Prediction [C]//Proceedings of the 2012 IEEE International Conference on Computer Science and Automation Engineering. Washington: IEEE Computer Society, 2012: 80-83.
[18]	ZHANG G, JIA S, LI X , et al. Weighted Score-level Feature Fusion Based on Dempster-Shafer Evidence Theory for Action Recognition[J]. Journal of Electronic Imaging, 2018,27(1):013021. doi: 10.1117/1.JEI.27.1.013021
[19]	宋静, 张雪英, 孙颖 , 等. 基于模糊综合评价法的情感语音数据库的建立[J]. 现代电子技术, 2016,39(13):51-54. doi: 10.16652/j.issn.1004-373x.2016.13.013
	SONG Jing, ZHANG Xueying, SUN Ying , et al. Establishment of Emotional Speech Database Based on Fuzzy Comprehensive Evaluation Method[J]. Modern Electronics Technique, 2016,39(13):51-54. doi: 10.16652/j.issn.1004-373x.2016.13.013
[20]	畅江, 张雪英, 张奇萍 , 等. 不同语种及非言语情感声音的ERP研究[J]. 清华大学学报: 自然科学版, 2016,56(10):1131-1136.
	CHANG Jiang, ZHANG Xueying, ZHANG Qiping , et al. ERP Research on the Emotional Voice for Different Languages and Non-speech Utterances[J]. Journal of Tsinghua University: Science and Technology, 2016,56(10):1131-1136.
[21]	LI K, WU Y, NAN Y , et al. Hierarchical Multi-class Classification in Multimodal Spacecraft Data Using DNN and Weighted Support Vector Machine[J]. Neurocomputing, 2017,259:55-65. doi: 10.1016/j.neucom.2016.08.131

情感信号	特征类型	维数统计	特征
语音信号	基本特征	1~98	语速、平均过零率、能量、基频及共振峰、MFCC
	非线性几何特征	99~135	吸引子到圆心和标识线距离、吸引子连续轨迹到圆心距离的总长度
	非线性属性特征	136~140	Hurst 、最大Lyapunov指数
脑电信号	基本特征	141~152	功率谱熵
	非线性几何特征	153~596	吸引子到圆心和标识线距离、吸引子连续轨迹到圆心距离的总长度
	非线性属性特征	597~668	Hurst 、最大Lyapunov指数、近似熵

情感信号	情感类型				平均识别率
情感信号	悲伤	生气	高兴	中性	平均识别率
语音信号	71.43	85.71	85.71	78.57	80.36
脑电信号	96.00	75.75	76.25	66.75	78.69
特征融合	85.00	84.75	81.25	74.75	81.44

实验类型	特征类型	情感类型				平均识别率
实验类型	特征类型	悲伤	生气	高兴	中性	平均识别率
单类特征分类	非线性几何特征	85.50	81.50	82.25	81.75	82.75
	非线性属性特征	95.04	7.62	72.31	84.72	64.92
	基本特征	79.25	86.00	96.00	73.50	83.69
DS融合	非线性全局特征	83.75	82.50	81.50	80.55	82.08
二次决策	情感特征全集	81.75	85.75	97.00	83.00	86.88

Multimodal emotion recognition for the fusion of speech and EEG signals

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 21

Related Articles 11

Metrics

Comments

Recommended 10

[1]	MAO Zhaoyong,WANG Yichen,WANG Xin,SHEN Junge. Vehicle video surveillance and analysis system for the expressway [J]. Journal of Xidian University, 2021, 48(5): 178-189.
[2]	ZHANG Shuwei,LI Junmin. Human body detection algorithm in complex monitoring scenes [J]. Journal of Xidian University, 2021, 48(5): 68-77.
[3]	SONG Jianqiao,WANG Feng,NIU Jin,SHI Zezhou,MA Junhui. Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals [J]. Journal of Xidian University, 2021, 48(4): 159-167.
[4]	LIU Mingqian,LI Kunming,WANG Chuanchuan,ZHANG Shun. Method for comprehensive evaluation of in-pulse characteristics of radar emitter signals [J]. Journal of Xidian University, 2019, 46(5): 8-14.
[5]	BIE Xiude;LIU Hongbin;CHANG Faliang;PENG Zhiyong. Multi-target tracking method based on the adaptive fragment and multi-feature fusion [J]. Journal of Xidian University, 2017, 44(2): 151-157.
[6]	ZHANG Minqing;ZHANG Yan;LI Delong;LUO Peng. Blind steganalysis algorithm based new calibration for JPEG images [J]. J4, 2015, 42(5): 133-138.
[7]	SONG Changhe;LI Yunsong;NING Jifeng;MOU Yongqiang. Study of multi-feature fusion methods for distribution fields in object tracking [J]. J4, 2015, 42(4): 1-7.
[8]	WANG Lei;SHI Ya;JI Hongbing. Specific radar emitter identification using multiset canonical correlation analysis [J]. J4, 2013, 40(2): 164-171.
[9]	. Specific Radar Emitter Identification Using Multiset Canonical Correlation Analysis [J]. J4, 2013, 40(2): 164-171.
[10]	LI Zhipeng;MA Tianxiang;DU Lan;XU Danlei;LIU Hongwei;ZHANG Zijing. Multi-class classifier design for feature fusion in radar HRRP recognition [J]. J4, 2013, 40(1): 111-117.
[11]	LI Yuanzheng;LU Zhaoyang;LI Jing. Robust video object tracking algorithm based on multi-feature fusion [J]. J4, 2012, 39(4): 1-6+45.