融合语音信号和脑电信号的多模态情感识别

doi:10.19665/j.issn1001-2400.2019.01.023

摘要/Abstract

摘要：

为构造有效的情感识别系统,通过声音刺激分别诱发出高兴、悲伤、生气以及中性4种情感,并采集相应的语音信号和脑电信号。首先,利用相空间重构技术提取脑电信号和语音信号的非线性几何特征和非线性属性特征,并结合两者的基本特征分别实现情感识别;然后,通过构建基于限制玻尔兹曼机的特征融合算法,从特征层融合的角度实现多模态情感识别;最后,利用二次决策算法从决策融合的角度构建多模态情感识别系统。实验结果显示,从特征融合的角度构建的多模态情感识别系统相比语音信号和脑电信号情感整体识别率,分别提高1.08%和2.75%;从决策融合的角度构建的多模态情感识别系统相比语音信号和脑电信号情感整体识别率,分别提高6.52%和8.19%;决策融合相比特征融合构建的多模态情感识别系统整体识别效果更优。因此,融合语音信号和脑电信号等不同来源的情感数据可以构造出更有效的情感识别系统。

关键词: 语音信号, 脑电信号, 特征融合, 决策融合

Abstract:

To construct an effective emotion recognition system, the emotions of joy, sadness, anger and neutrality are induced by sound stimulation, and the corresponding speech and EEG signals are collected. First, this paper extracts the nonlinear geometric feature and nonlinear attribute feature of EEG and speech signals by phase space reconstruction respectively, and the emotion recognition is realized by combining the basic features. Then, a feature fusion algorithm based on the Restricted Boltzmann Machine is constructed to realize multimodal emotion recognition from the perspective of feature fusion. Finally, a multimodal emotion recognition system is constructed through decision fusion by using the quadratic decision algorithm. The results show that the overall recognition rate of the multimodal emotion recognition system constructed by feature fusion is 1.08% and 2.75% higher than that of speech signals and that of EEG signals respectively, and that the overall recognition rate of the multimodal emotion recognition system constructed by decision fusion is 6.52% and 8.19% higher than that of speech signals and that of EEG signals respectively. The overall recognition effect of the multimodal emotion recognition system based on decision fusion is better than that of feature fusion. A more effective emotion recognition system can be constructed by combining the emotional data of different channels such as speech signals and EEG signals.

Key words: speech signals, electroencephalo-graph signals, feature fusion, decision fusion

中图分类号:

TP391.4

马江河,孙颖,张雪英. 融合语音信号和脑电信号的多模态情感识别[J]. 西安电子科技大学学报, 2019, 46(1): 143-150.

MA Jianghe,SUN Ying,ZHANG Xueying. Multimodal emotion recognition for the fusion of speech and EEG signals[J]. Journal of Xidian University, 2019, 46(1): 143-150.

图/表 9

图1

表1

图2

图3

图4

表2

图5

表3

图6

参考文献 21

[1]	ZHANG S, ZHANG S, HUANG T , et al. Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching[J]. IEEE Transactions on Multimedia, 2018,20(6):1576-1590. doi: 10.1109/TMM.2017.2766843
[2]	黄程韦, 金赟, 王青云 , 等. 基于语音信号与心电信号的多模态情感识别[J]. 东南大学学报:自然科学版, 2010,40(5):895-900. doi: 10.3969/j.issn.1001-0505.2010.05.003
	HUANG Chengwei, JIN Yun, WANG Qingyun , et al. Multimodal Emotion Recognition Based on Speech and ECG Signals[J]. Journal of Southeast University: Natural Science Edition, 2010,40(5):895-900. doi: 10.3969/j.issn.1001-0505.2010.05.003
[3]	PEREZ-GASPAR L A, CABALLERO-MORALES S O, TRUJILLO-ROMERO F . Multimodal Emotion Recognition with Evolutionary Computation for Human-robot Interaction[J]. Expert Systems with Applications, 2016,66:42-61. doi: 10.1016/j.eswa.2016.08.047
[4]	HUANG X, KORTELAINEN J, ZHAO G , et al. Multi-modal Emotion Analysis from Facial Expressions and Electroencephalogram[J]. Computer Vision and Image Understanding, 2016,147:114-124. doi: 10.1016/j.cviu.2015.09.015
[5]	刘仁怡 . 基于脑电和语音信号的情感识别研究[D]. 天津: 天津师范大学, 2015.
[6]	姚慧, 孙颖, 张雪英 . 情感语音的非线性动力学特征[J]. 西安电子科技大学学报, 2016,43(5):167-172. doi: 10.3969/j.issn.1001-2400.2016.05.029
	YAO Hui, SUN Ying, ZHANG Xueying . Research on Nonlinear Dynamics Features of Emotional Speech[J]. Journal of Xidian University, 2016,43(5):167-172. doi: 10.3969/j.issn.1001-2400.2016.05.029
[7]	HATAMIKIA S, NASRABADI A M. Recognition of Emotional States Induced by Music Videos Based on Nonlinear Feature Extraction and SOM Classification [C]//Proceedings of the 2014 21st Iranian Conference on Biomedical Engineering. Piscataway: IEEE, 2014: 333-337.
[8]	孙颖, 宋春晓 . 相空间重构的情感语音特征提取及优化[J]. 西安电子科技大学学报, 2017,44(6):162-168. doi: 10.3969/j.issn.1001-2400.2017.06.028
	SUN Ying, SONG Chunxiao . Emotional Speech Feature Extraction and Optimization of Phase Space Reconstruction[J]. Journal of Xidian University, 2017,44(6):162-168. doi: 10.3969/j.issn.1001-2400.2017.06.028
[9]	TAKENS F. Detecting Strange Attractors in Turbulence [C]//Lecture Notesin Mathematics: 898. Heidelberg: Springer-Verlag, 1981: 366-381.
[10]	LAHMIRI S . Generalized Hurst Exponent Estimates Differentiate EEG Signals of Healthy and Epileptic Patients[J]. Physica A: Statistical Mechanics and Its Applications, 2018,490:378-385. doi: 10.1016/j.physa.2017.08.084
[11]	KORDA A I, ASVESTAS P A, MATSOPOULOS G K , et al. Automatic Identification of Eye Movements Using the Largest Lyapunov Exponent[J]. Biomedical Signal Processing and Control, 2018,41:10-20. doi: 10.1016/j.bspc.2017.11.004
[12]	WOLF A, SWIFT J B, SWINNEY H L , et al. Determining Lyapunov Exponents from a Time Series[J]. Physica D: Nonlinear Phenomena, 1985,16(3):285-317. doi: 10.1016/0167-2789(85)90011-9
[13]	DAVID G F, PAU M M, JORGE J N , et al. Noisy EEG Signals Classification Based on Entropymetrics. Performance Assessment Using First and Second Generation Statistics[J]. Computers in Biology and Medicine, 2017,87:141-151. doi: 10.1016/j.compbiomed.2017.05.028 pmid: 28595129
[14]	HARRINGTON P D B . Feature Expansion by a Continuous Restricted Boltzmann Machine for Near-infrared Spectrometric Calibration[J]. Analytica Chimica Acta, 2018,1010:20-28. doi: 10.1016/j.aca.2018.01.026
[15]	ZHENG H, ZHANG S, SUN X. Classification Recognition of Anchor Rod Based on PSO-SVM [C]//Proceedings of the 2017 29th Chinese Control and Decision Conference. Piscataway: IEEE, 2017: 2207-2212.
[16]	TENG K, WANG J. Classification Related Manifold Dimension Estimation with Restricted Boltzmann Machine [C]//Proceedings of the 2013 7th International Conference on Image and Graphics. Washington: IEEE Computer Society, 2013: 857-862.
[17]	CAI X, HU S, LIN X. Feature Extraction Using Restricted Boltzmann Machine for Stock Price Prediction [C]//Proceedings of the 2012 IEEE International Conference on Computer Science and Automation Engineering. Washington: IEEE Computer Society, 2012: 80-83.
[18]	ZHANG G, JIA S, LI X , et al. Weighted Score-level Feature Fusion Based on Dempster-Shafer Evidence Theory for Action Recognition[J]. Journal of Electronic Imaging, 2018,27(1):013021. doi: 10.1117/1.JEI.27.1.013021
[19]	宋静, 张雪英, 孙颖 , 等. 基于模糊综合评价法的情感语音数据库的建立[J]. 现代电子技术, 2016,39(13):51-54. doi: 10.16652/j.issn.1004-373x.2016.13.013
	SONG Jing, ZHANG Xueying, SUN Ying , et al. Establishment of Emotional Speech Database Based on Fuzzy Comprehensive Evaluation Method[J]. Modern Electronics Technique, 2016,39(13):51-54. doi: 10.16652/j.issn.1004-373x.2016.13.013
[20]	畅江, 张雪英, 张奇萍 , 等. 不同语种及非言语情感声音的ERP研究[J]. 清华大学学报: 自然科学版, 2016,56(10):1131-1136.
	CHANG Jiang, ZHANG Xueying, ZHANG Qiping , et al. ERP Research on the Emotional Voice for Different Languages and Non-speech Utterances[J]. Journal of Tsinghua University: Science and Technology, 2016,56(10):1131-1136.
[21]	LI K, WU Y, NAN Y , et al. Hierarchical Multi-class Classification in Multimodal Spacecraft Data Using DNN and Weighted Support Vector Machine[J]. Neurocomputing, 2017,259:55-65. doi: 10.1016/j.neucom.2016.08.131

情感信号	特征类型	维数统计	特征
语音信号	基本特征	1~98	语速、平均过零率、能量、基频及共振峰、MFCC
	非线性几何特征	99~135	吸引子到圆心和标识线距离、吸引子连续轨迹到圆心距离的总长度
	非线性属性特征	136~140	Hurst 、最大Lyapunov指数
脑电信号	基本特征	141~152	功率谱熵
	非线性几何特征	153~596	吸引子到圆心和标识线距离、吸引子连续轨迹到圆心距离的总长度
	非线性属性特征	597~668	Hurst 、最大Lyapunov指数、近似熵

情感信号	情感类型				平均识别率
情感信号	悲伤	生气	高兴	中性	平均识别率
语音信号	71.43	85.71	85.71	78.57	80.36
脑电信号	96.00	75.75	76.25	66.75	78.69
特征融合	85.00	84.75	81.25	74.75	81.44

实验类型	特征类型	情感类型				平均识别率
实验类型	特征类型	悲伤	生气	高兴	中性	平均识别率
单类特征分类	非线性几何特征	85.50	81.50	82.25	81.75	82.75
	非线性属性特征	95.04	7.62	72.31	84.72	64.92
	基本特征	79.25	86.00	96.00	73.50	83.69
DS融合	非线性全局特征	83.75	82.50	81.50	80.55	82.08
二次决策	情感特征全集	81.75	85.75	97.00	83.00	86.88