一种面向时空神经网络的潜在情绪识别方法

doi:10.19665/j.issn1001-2400.2021.04.021

摘要/Abstract

摘要：

微表情对情绪识别有着一定的作用,但在人为隐藏的情况下则容易出现误判。生理信号的识别效果虽然较为准确,但其数据的采集往往复杂,不便于用于快速的人员情绪检测中。针对上述问题,采用非接触基于色度模型的方式采集脉搏信号,并根据脉搏信号提取特征,融合提出的时空神经网络实现潜在情绪识别,平均识别率约78.59%和76.91%。实验结果表明,提出的双路潜在情绪识别框架可以很好地融合微表情和生理信号中所包含的情绪信息,在微表情识别中的效果较现阶段常用的微表情识别算法的效果有一定提升。

关键词: 潜在情绪识别, 色度模型, 人脸脉搏信号, 深度学习, 神经网络, 决策融合

Abstract:

(Micro) expression has a certain effect on emotion recognition,but in the case of artificial concealment,it is prone to misjudgment.Although the recognition effect of physiological signals is more accurate,the data collection is often complicated,which is not convenient for rapid personnel emotion checking.In response to the above problem,this paper adopts a non-contact chromatic model-based method to collect pulse signals,extract features based on the pulse signals,and integrate the proposed spatio-temporal neural network to realize potential emotion recognition.Experimental results show that the proposed two-way latent emotion recognition framework can well integrate the emotion information contained in micro-expressions and physiological signals,and that the effect in micro-expression recognition is improved to some extent compared with the current micro-expression recognition algorithms commonly used at this stage.

Key words: latent emotion recognition, chroma model, deeplearning, facial pulse signal, neural networks, decision fusion

中图分类号:

TP183

宋剑桥,王峰,牛锦,师泽洲,马军辉. 一种面向时空神经网络的潜在情绪识别方法[J]. 西安电子科技大学学报, 2021, 48(4): 159-167.

SONG Jianqiao,WANG Feng,NIU Jin,SHI Zezhou,MA Junhui. Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals[J]. Journal of Xidian University, 2021, 48(4): 159-167.

图/表 12

图1

图2

图3

图4

表1

图5

表2

图6

表3

表4

表5

表6

参考文献 25

[1]	MUKHERJEE S, VAMSHI B, REDDY K V S V K, et al. Recognizing Facial Expressions Using Novel Motion Based Features[C]//Proceedings of the Tenth Indian Conference on Computer Vision,Graphics and Image Processing.Piscataway:IEEE, 2016: 32.
[2]	LI X, HONG X, MOILANEN A, et al. Towards Reading Hidden Emotions:A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods[J]. IEEE Transactions on Affective Computing, 2018, 9(4):563-577. doi: 10.1109/T-AFFC.5165369
[3]	WU Q, SHEN X B, FU X L. The Machine Knows What You Are Hiding:An Automatic Micro-Expression Recognition System[C]//Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction-Volume Part II.Springer Berlin, 2011:152-162.
[4]	LU H, KPALMA K, RONSIN J. Motion Descriptors for Micro-Expression Recognition[J]. Signal Processing-Image Communication, 2018, 67:108-117. doi: 10.1016/j.image.2018.05.014
[5]	LIU Y J, ZHANG J K, YAN W J, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/T-AFFC.5165369
[6]	LI Q Y, YU J, KURIHARA T, et al. Micro-Expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow[C]//Proceedings of the 2018 5th International Conference on Control,Decision and Information Technologies.Piscataway:IEEE, 2018:265-270.
[7]	PENG M, WU Z, ZHANG Z H, et al. From Macro to Micro Expression Recognition:Deep Learning on Small Datasets Using Transfer Learning[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 2018:657-661.
[8]	WANG S J, LI B J, LIU Y J, et al. Micro-Expression Recognition with Small Sample Size by Transferring Long-Term Convolutional Neural Network[J]. Neurocomputing, 2018, 312:251-262. doi: 10.1016/j.neucom.2018.05.107
[9]	KIM D H, BADDAR W J, RO Y M. Micro-expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations[C]//Proceedings of the 2016 ACM Multimedia Conference.Amsterdam:ACM, 2016:382-386.
[10]	KIM D H, BADDAR W, JANG J, et al. Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition[J]. IEEE Transactions on Affective Computing, 2019, 10(2):223-236. doi: 10.1109/T-AFFC.5165369
[11]	KHOR H Q, SEE J, PHAN R C W, et al. Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 2018:667-674.
[12]	FRANTZIDIS C A, BRATSAS C, PAPADELIS C L, et al. Toward Emotion Aware Computing:an Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli[J]. IEEE Transactions on Information Technology in Biomedicine, 2010, 14(3):589-597. doi: 10.1109/TITB.2010.2041553
[13]	KHEZRI M, FIROOZABADI M, SHARAFAT A R. Reliable Emotion Recognition System Based on Dynamic Adaptive Fusion of Forehead Biopotentials and Physiological Signals[J]. Computer Methods and Programs in Biomedicine, 2015, 122(2):149-164. doi: 10.1016/j.cmpb.2015.07.006
[14]	TORRES-VALENCIA C, ALVAREZ-LOPEZ M, OROZCO-GUTIERREZ A. SVM-Based Feature Selection Methods for Emotion Recognition from Multimodal Data[J]. Journal on Multimodal User Interfaces, 2017, 11(1):9-23. doi: 10.1007/s12193-016-0222-y
[15]	DUC B, BIGUN E S, BIGUN J, et al. Fusion of Audio and Video Information for Multi Modal Person Authentication[J]. Pattern Recognition Letters, 1997, 18(9):835-843. doi: 10.1016/S0167-8655(97)00071-8
[16]	BAILENSON J N, PONTIKAKIS E D, MAUSS I B, et al. Real-Time Classification of Evoked Emotions Using Facial Feature Tracking and Physiological Responses[J]. International Journal of Human Computer Studies, 2008, 66(5):303-317. doi: 10.1016/j.ijhcs.2007.10.011
[17]	TRIPATHI S, BEIGI H. Multi-modal Emotion Recognition on IEMOCAP Dataset Using Deep Learning[J/OL].[2020-03-20].https://arxiv.org/pdf/1804.05788.pdf .
[18]	ZHANG D, WU L, LI S, et al. Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions[C]//Proceedings of the 2019 IEEE International Conference on Multimedia and Expo.Piscataway:IEEE Computer Society, 2019:724-729.
[19]	VIOLA P, JONES M. Robust Real-Time Object Detection[J]. International Journal of Computer Vision, 2001, 57:137-154. doi: 10.1023/B:VISI.0000013087.49260.fb
[20]	LIBERZON D, TEMPO R. Common Lyapunov Functions and Gradient Algorithms[J]. IEEE Transactions on Automatic Control, 2004, 49(6):990-994. doi: 10.1109/TAC.2004.829632
[21]	COSTA M, GOLDBERGER A, PENG C K. Multiscale Entropy Analysis of Biological Signals[J]. Physical Review E, 2005, 71(2):21906. doi: 10.1103/PhysRevE.71.021906
[22]	NITISH S, HINTON G, KRIZHEVSKY A, et al. Dropout:a Simple Way to Prevent Neural Networks from Overfitting[J]. Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[23]	HUANG X H, ZHAO G Y, HONG X P, et al. Spontaneous Facial Micro-Expression Analysis Using Spatiotemporal Completed Local Quantized Patterns[J]. Neurocomputing, 2016, 175:564-578. doi: 10.1016/j.neucom.2015.10.096
[24]	LI J, WANG Y D, JOHN S, et al. Micro-Expression Recognition Based on 3D Flow Convolutional Neural Network[J]. Pattern Analysis and Applications, 2019, 22(4):1331-1339. doi: 10.1007/s10044-018-0757-5
[25]	LIU Y J, ZHANG J K, YAN W J, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/T-AFFC.5165369

网络层名称	卷积核个数	卷积核尺寸	步长	输出维度
输入				64×64×96
Conv3D_1	32	3×3×15	1×1×1	32×62×62×82
Conv3D_2	32	3×3×15	1×1×1	32×60×60×68
Maxpooling3D		3×3×3	1×1×1	32×20×20×22
ConvLSTM_1	16	3×3	1×1×1	32×16×18×20
ConvLSTM_2	16	3×3	1×1×1	32×16×16×18
Flatten				147 456
FC_1				128

情绪	CAS(ME)^2	USPE
生气	275=220+55	515=412+103
厌恶	370=296+74	435=348+87
开心	655=524+131	600=480+120
惊讶	180=144+36
悲伤		490=392+98
总计	1 480=1 184+296	2 040=1 632+408

卷积核尺寸		不同尺寸3D卷积核的识别率/%
Conv3D_1	Conv3D_2	CAS(ME)^2	TYUTPE
3×3×7	3×3×7	69.83	70.26
3×3×15	3×3×15	75.78	73.92
3×3×19	3×3×19	46.36	51.23
5×5×7	5×5×7	69.49	70.52
5×5×15	5×5×15	68.67	67.31
5×5×19	5×5×19	48.92	47.36

卷积核尺寸		不同尺寸3D卷积核的识别率/%
ConvLSTM_1	ConvLSTM_2	CAS(ME)^2	TYUTPE
2×2	2×2	69.95	71.01
3×3	3×3	77.02	74.83
4×4	4×4	56.20	58.35
5×5	5×5	54.73	60.53

实验	CAS(ME)^2数据库情绪识别率/%					USPE数据库情绪识别率/%
实验	生气	厌恶	开心	惊讶	平均	生气	厌恶	开心	悲伤	平均
实验1	58.12	59.45	67.76	71.93	64.31	57.88	59.10	75.96	62.90	64.04
实验2	67.52	76.94	83.23	80.39	77.02	73.45	78.62	81.83	65.42	74.83
实验3	73.20	74.13	81.58	85.43	78.59	68.59	76.51	88.62	73.93	76.91