一种面向时空神经网络的潜在情绪识别方法

doi:10.19665/j.issn1001-2400.2021.04.021

Abstract

Abstract:

(Micro) expression has a certain effect on emotion recognition,but in the case of artificial concealment,it is prone to misjudgment.Although the recognition effect of physiological signals is more accurate,the data collection is often complicated,which is not convenient for rapid personnel emotion checking.In response to the above problem,this paper adopts a non-contact chromatic model-based method to collect pulse signals,extract features based on the pulse signals,and integrate the proposed spatio-temporal neural network to realize potential emotion recognition.Experimental results show that the proposed two-way latent emotion recognition framework can well integrate the emotion information contained in micro-expressions and physiological signals,and that the effect in micro-expression recognition is improved to some extent compared with the current micro-expression recognition algorithms commonly used at this stage.

Key words: latent emotion recognition, chroma model, deeplearning, facial pulse signal, neural networks, decision fusion

CLC Number:

TP183

SONG Jianqiao,WANG Feng,NIU Jin,SHI Zezhou,MA Junhui. Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals[J].Journal of Xidian University, 2021, 48(4): 159-167.

Figures/Tables 12

References 25

[1]	MUKHERJEE S, VAMSHI B, REDDY K V S V K, et al. Recognizing Facial Expressions Using Novel Motion Based Features[C]//Proceedings of the Tenth Indian Conference on Computer Vision,Graphics and Image Processing.Piscataway:IEEE, 2016: 32.
[2]	LI X, HONG X, MOILANEN A, et al. Towards Reading Hidden Emotions:A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods[J]. IEEE Transactions on Affective Computing, 2018, 9(4):563-577. doi: 10.1109/T-AFFC.5165369
[3]	WU Q, SHEN X B, FU X L. The Machine Knows What You Are Hiding:An Automatic Micro-Expression Recognition System[C]//Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction-Volume Part II.Springer Berlin, 2011:152-162.
[4]	LU H, KPALMA K, RONSIN J. Motion Descriptors for Micro-Expression Recognition[J]. Signal Processing-Image Communication, 2018, 67:108-117. doi: 10.1016/j.image.2018.05.014
[5]	LIU Y J, ZHANG J K, YAN W J, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/T-AFFC.5165369
[6]	LI Q Y, YU J, KURIHARA T, et al. Micro-Expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow[C]//Proceedings of the 2018 5th International Conference on Control,Decision and Information Technologies.Piscataway:IEEE, 2018:265-270.
[7]	PENG M, WU Z, ZHANG Z H, et al. From Macro to Micro Expression Recognition:Deep Learning on Small Datasets Using Transfer Learning[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 2018:657-661.
[8]	WANG S J, LI B J, LIU Y J, et al. Micro-Expression Recognition with Small Sample Size by Transferring Long-Term Convolutional Neural Network[J]. Neurocomputing, 2018, 312:251-262. doi: 10.1016/j.neucom.2018.05.107
[9]	KIM D H, BADDAR W J, RO Y M. Micro-expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations[C]//Proceedings of the 2016 ACM Multimedia Conference.Amsterdam:ACM, 2016:382-386.
[10]	KIM D H, BADDAR W, JANG J, et al. Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition[J]. IEEE Transactions on Affective Computing, 2019, 10(2):223-236. doi: 10.1109/T-AFFC.5165369
[11]	KHOR H Q, SEE J, PHAN R C W, et al. Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 2018:667-674.
[12]	FRANTZIDIS C A, BRATSAS C, PAPADELIS C L, et al. Toward Emotion Aware Computing:an Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli[J]. IEEE Transactions on Information Technology in Biomedicine, 2010, 14(3):589-597. doi: 10.1109/TITB.2010.2041553
[13]	KHEZRI M, FIROOZABADI M, SHARAFAT A R. Reliable Emotion Recognition System Based on Dynamic Adaptive Fusion of Forehead Biopotentials and Physiological Signals[J]. Computer Methods and Programs in Biomedicine, 2015, 122(2):149-164. doi: 10.1016/j.cmpb.2015.07.006
[14]	TORRES-VALENCIA C, ALVAREZ-LOPEZ M, OROZCO-GUTIERREZ A. SVM-Based Feature Selection Methods for Emotion Recognition from Multimodal Data[J]. Journal on Multimodal User Interfaces, 2017, 11(1):9-23. doi: 10.1007/s12193-016-0222-y
[15]	DUC B, BIGUN E S, BIGUN J, et al. Fusion of Audio and Video Information for Multi Modal Person Authentication[J]. Pattern Recognition Letters, 1997, 18(9):835-843. doi: 10.1016/S0167-8655(97)00071-8
[16]	BAILENSON J N, PONTIKAKIS E D, MAUSS I B, et al. Real-Time Classification of Evoked Emotions Using Facial Feature Tracking and Physiological Responses[J]. International Journal of Human Computer Studies, 2008, 66(5):303-317. doi: 10.1016/j.ijhcs.2007.10.011
[17]	TRIPATHI S, BEIGI H. Multi-modal Emotion Recognition on IEMOCAP Dataset Using Deep Learning[J/OL].[2020-03-20].https://arxiv.org/pdf/1804.05788.pdf .
[18]	ZHANG D, WU L, LI S, et al. Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions[C]//Proceedings of the 2019 IEEE International Conference on Multimedia and Expo.Piscataway:IEEE Computer Society, 2019:724-729.
[19]	VIOLA P, JONES M. Robust Real-Time Object Detection[J]. International Journal of Computer Vision, 2001, 57:137-154. doi: 10.1023/B:VISI.0000013087.49260.fb
[20]	LIBERZON D, TEMPO R. Common Lyapunov Functions and Gradient Algorithms[J]. IEEE Transactions on Automatic Control, 2004, 49(6):990-994. doi: 10.1109/TAC.2004.829632
[21]	COSTA M, GOLDBERGER A, PENG C K. Multiscale Entropy Analysis of Biological Signals[J]. Physical Review E, 2005, 71(2):21906. doi: 10.1103/PhysRevE.71.021906
[22]	NITISH S, HINTON G, KRIZHEVSKY A, et al. Dropout:a Simple Way to Prevent Neural Networks from Overfitting[J]. Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[23]	HUANG X H, ZHAO G Y, HONG X P, et al. Spontaneous Facial Micro-Expression Analysis Using Spatiotemporal Completed Local Quantized Patterns[J]. Neurocomputing, 2016, 175:564-578. doi: 10.1016/j.neucom.2015.10.096
[24]	LI J, WANG Y D, JOHN S, et al. Micro-Expression Recognition Based on 3D Flow Convolutional Neural Network[J]. Pattern Analysis and Applications, 2019, 22(4):1331-1339. doi: 10.1007/s10044-018-0757-5
[25]	LIU Y J, ZHANG J K, YAN W J, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/T-AFFC.5165369

网络层名称	卷积核个数	卷积核尺寸	步长	输出维度
输入				64×64×96
Conv3D_1	32	3×3×15	1×1×1	32×62×62×82
Conv3D_2	32	3×3×15	1×1×1	32×60×60×68
Maxpooling3D		3×3×3	1×1×1	32×20×20×22
ConvLSTM_1	16	3×3	1×1×1	32×16×18×20
ConvLSTM_2	16	3×3	1×1×1	32×16×16×18
Flatten				147 456
FC_1				128

情绪	CAS(ME)^2	USPE
生气	275=220+55	515=412+103
厌恶	370=296+74	435=348+87
开心	655=524+131	600=480+120
惊讶	180=144+36
悲伤		490=392+98
总计	1 480=1 184+296	2 040=1 632+408

卷积核尺寸		不同尺寸3D卷积核的识别率/%
Conv3D_1	Conv3D_2	CAS(ME)^2	TYUTPE
3×3×7	3×3×7	69.83	70.26
3×3×15	3×3×15	75.78	73.92
3×3×19	3×3×19	46.36	51.23
5×5×7	5×5×7	69.49	70.52
5×5×15	5×5×15	68.67	67.31
5×5×19	5×5×19	48.92	47.36

卷积核尺寸		不同尺寸3D卷积核的识别率/%
ConvLSTM_1	ConvLSTM_2	CAS(ME)^2	TYUTPE
2×2	2×2	69.95	71.01
3×3	3×3	77.02	74.83
4×4	4×4	56.20	58.35
5×5	5×5	54.73	60.53

实验	CAS(ME)^2数据库情绪识别率/%					USPE数据库情绪识别率/%
实验	生气	厌恶	开心	惊讶	平均	生气	厌恶	开心	悲伤	平均
实验1	58.12	59.45	67.76	71.93	64.31	57.88	59.10	75.96	62.90	64.04
实验2	67.52	76.94	83.23	80.39	77.02	73.45	78.62	81.83	65.42	74.83
实验3	73.20	74.13	81.58	85.43	78.59	68.59	76.51	88.62	73.93	76.91

Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 25

Related Articles 15

Metrics

Comments

Recommended 0

方法	CAS(ME)^2识别率	USPE识别率
LBP-TOP	68.46	66.59
STCLQP^[23]	69.02	67.30
3D-FCNN^[24]	72.06	71.36
MDMO^[25]	65.23	63.35
CNN with Augmentation^[17]	74.32	72.11
文中方法	78.59	76.91

[1]	QU Jiahui, HE Jie, DONG Wenqian, LI Yunsong, ZHANG Tongzhen, YANG Yufei. Change detection method based on multi-scale and multi-resolution information fusion [J]. Journal of Xidian University, 2025, 52(1): 105-116.
[2]	KONG Fanqiang, YU Shengjie, WANG Kun, FANG Xu, LV Zhijie. Hyperspectral image unmixing method based on convolutional recurrent neural networks [J]. Journal of Xidian University, 2025, 52(1): 142-151.
[3]	LI Linke, CHEN Jie, LIU Jun. Improved schemes and applications of the neural network differential distinguisher [J]. Journal of Xidian University, 2025, 52(1): 196-214.
[4]	WANG Danyang, PIAO Chunying, LIU Qi, GUAN Lei, LI Zan. CTS features based electromagnetic interference identification at radio observatory site [J]. Journal of Xidian University, 2025, 52(1): 80-93.
[5]	WANG Yulai, LIAO Xiaomin, HE Haiguang, YE Guojun. Knowledge graph assisted spectrum resource optimization algorithm for UAVs [J]. Journal of Xidian University, 2024, 51(5): 58-70.
[6]	LI Xiaohan, YANG Yanbo, ZHANG Jiawei, LI Baoshan, MA Jianfeng. Graph neural network vulnerability detection for ethernet smart contracts [J]. Journal of Xidian University, 2024, 51(4): 139-150.
[7]	ZHANG Xiaohong, XIANG Shijun, HUANG Hongbin. Hiding images in audio based on invertible neural networks [J]. Journal of Xidian University, 2024, 51(4): 226-238.
[8]	GONG Junyang, FU Weihong, FANG Houzhang. Research on lightweight and feature enhancement of SAR image ship targets detection [J]. Journal of Xidian University, 2024, 51(2): 96-106.
[9]	ZHONG Hao, BIAN Shan, WANG Chuntao. Real world image tampering localization combining the self-attention mechanism and convolutional neural networks [J]. Journal of Xidian University, 2024, 51(1): 135-146.
[10]	ZHANG Xinyu, LIANG Yu, ZHANG Wei. Real-time smoke segmentation algorithm combining global and local information [J]. Journal of Xidian University, 2024, 51(1): 147-156.
[11]	SHEN Lu, SU Hongtao, WANG Jin, MAO Zhi, JING Xinchen, LI Ze. Attention autocorrelation mechanism-based residual clutter suppression method [J]. Journal of Xidian University, 2024, 51(1): 41-51.
[12]	HUANG Heyuan, MU Caihong, FANG Yunfei, LIU Yi. Graph convolution neural network for recommendation using graph negative sampling [J]. Journal of Xidian University, 2024, 51(1): 86-99.
[13]	DU Mingyang, DU Meng, PAN Jifei, BI Daping. Generative adversarial model for radar intra-pulse signal denoising and recognition [J]. Journal of Xidian University, 2023, 50(6): 133-147.
[14]	DONG Meng,GAO Yiming,PAN Weitao,QIU Zhiliang,YANG Jianlei,DI Zhixiong,ZHENG Ling. RELIC-GNN:an efficient state register identification algorithm [J]. Journal of Xidian University, 2023, 50(3): 142-150.
[15]	ZHANG Zehuan, LIU Qiang, GUO Difei. High efficient framework for large-scale zero-shot image recognition [J]. Journal of Xidian University, 2022, 49(6): 103-110.