一种基于局部表征的面部表情识别算法

doi:10.19665/j.issn1001-2400.2021.05.013

Abstract

Abstract:

Expression is an important embodiment of human inner emotion change.Current expression recognition methods usually rely on global facial features,ignoring local features extraction.Psychologists point out that different facial expressions correspond to different regions of local muscle movement.In this paper,we propose an expression recognition algorithm based on local representation,referred to as EAU-CNN.In order to extract the local features of the face,the whole face image is first divided into 43 sub-regions according to the 68 feature points of the face.Then,8 local candidate regions covered by the muscle motion region and the facial salient organs are selected as the input of the convolution neural network.In order to balance the features of local candidate regions,the EAU-CNN adopts 8 parallel feature extraction branches,each of which dominates the full connected layer of different dimensions.The outputs of the branches are adaptively connected in terms of attention to highlight the importance of different local candidate regions.Finally,the expressions are divided into 7 categories:neutral,angry,disgusted,surprised,happy,sad and afraid by the Softmax function.In this paper,the algorithm is evaluated on CK +,JAFFE and custom FED datasets.The average accuracy of the proposed algorithm is 99.85%,96.61% and 98.6%,respectively.The evaluation index is 6.01%,10.17%,6.09% higher than that of the S-Patches algorithm.The results show that local representation can improve the performance of emotional recognition.

Key words: expression recognition, motion unit partition, convolutional neural network, loss function

CLC Number:

TP183

CHEN Changchuan,WANG Haining,HUANG Lian,HUANG Tao,LI Lianjie,HUANG Xiangkang,DAI Shaosheng. Facial expression recognition based on local representation[J].Journal of Xidian University, 2021, 48(5): 100-109.

Figures/Tables 18

References 22

[1]	徐峰, 张军平. 人脸微表情识别综述[J]. 自动化学报, 2017, 43(3):333-348.
	XU Feng, ZHANG Junping. Facial Expression Recognition:A Survey[J]. Acta Automatica Sinica, 2017, 43(3):333-348.
[2]	MEHRABIAN A. Communication without Words[J]. Psychology Today, 1968, 2(4):53-56.
[3]	EKMAN P, FRIESEN W V. Facial Action Coding System:A Technique for the Measurement of Facial Movement[M]. Palo Alto: Consulting Psychologists Press, 1978.
[4]	PANTIC M, PATRAS I. Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J]. IEEE Transactions on Systems,Man,and Cybernetics,Part B (Cybernetics), 2006, 36(2):433-449. doi: 10.1109/TSMCB.2005.859075
[5]	COHEN I, SEBE N, GARG A, et al. Facial Expression Recognition from Video Sequences:Temproal and Static Modeling[J]. Computer Vision and Image Understanding, 2003, 91(1-2):160-187. doi: 10.1016/S1077-3142(03)00081-X
[6]	邓捷方, 杨勇, 王国胤. 一种基于局部特征融合的表情识别方法[J]. 重庆邮电大学学报(自然科学版), 2011, 23(5):626-630.
	DENG Jiefang, YANG Yong, WANG Guoyin. An Expression Recognition Method Based on Local Feature Fusion[J]. Journal of Chongqing University of Posts and Telecommunications, 2011, 23(5):626-630.
[7]	杨勇, 蔡舒博. 一种基于两步降维和并行特征融合的表情识别方法[J]. 重庆邮电大学学报(自然科学版), 2015, 27(3):377-385.
	YANG Yong, CAI Shubo. Facial Expression Recognition Method Based on Two-Steps Dimensionality Reduction and Parallel Feature Fusion[J]. Journal of Chongqing University of Posts and Telecommunications, 2015, 27(3):377-385.
[8]	LAJEVARDI S M, HUSSAIN Z M. Automatic Facial Expression Recognition:Feature Extraction and Selection[J]. Signal,Image and Video Processing, 2012, 6:159-169. doi: 10.1007/s11760-010-0177-5
[9]	LUCEY P, COHN J F, KANADE T, et al. The Extended Cohn-Kanade Dataset(CK+):A Complete Dataset for Action Unit and Emotion-Specified Expression[C]// Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops.Piscataway:IEEE, 2010:94-101.
[10]	LYONS M, AKAMATSU S, KAMACHI M, et al. Coding Facial Expressions with Gabor Wavelets[C]// Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 1998:1-6.
[11]	YANG B, CAO J, NI R, et al. Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images[J]. IEEE Access, 2018, 6:4630-4640. doi: 10.1109/ACCESS.2017.2784096
[12]	MOLLAHOSSEINI A, CHAN D, MAHOOR M H, et al. Going Deeper in Facial Expression Recognition Using Deep Neural Networks[C]// 2016 IEEE Winter Conference on Applications of Computer Vision.Piscataway:IEEE, 2016:1-10.
[13]	KAHOUS E, PAL C, BOUTHILLIER X, et al. Combing Modality Specific Deep Neural Networks for Emotion Recognition in Video[C]// Proceedings of the 15th ACM on International Conference on Multimodal Interaction.New York:ACM, 2013:543-550.
[14]	HINTON G, DENG L, YU D, et al. Deep Neural Networks for Acoustic Modeling in Speech Recognition:The Shared Views of Four Research Groups[J]. IEEE Signal Processing Magazine, 2012, 29(6):82-97.
[15]	冯晓毅, 赖阳明, 王文星等. 一种边缘二进制码的表情识别方法[J]. 西安电子科技大学学报, 2015, 42(3):186-191.
	FENG Xiaoyi, LAI Yangming, WANG Wenxing, et al. Facial Expression Recognition Based on the Binary Code of Edges[J]. Journal of Xidian University, 2015, 42(3):186-191.
[16]	谭小慧, 李昭伟, 樊亚春. 基于多尺度细节增强的面部表情识别方法[J]. 电子与信息学报, 2019, 41(11):2752-2759.
	TAN Xiaohui, LI Zhaowei, FAN Yachun. Facial Expression Recognition Method Based on Multi-Scale Detail Enhancement[J]. Journal of Electronics & Information Technology, 2019, 41(11):2572-2759.
[17]	HAPPYS L, ROUTRAY A. Automatic Facial Expression Recognition Using Features of Salient Facial Patches[J]. IEEE Transactions on Affective Computing, 2015, 6(1):1-12. doi: 10.1109/TAFFC.2014.2386334
[18]	KALEEKAL T, SINGH J. Facial Expression Recognition Using Hahn Moment on Facial Patches[C]// 2019 4th International Conference on Recent Trends on Electronics,Information,Communication & Technology (RTEICT).Piscataway:IEEE, 2019:1338-1344.
[19]	LI Y, WANG S, ZHAO Y, et al. Simultaneous Facial Feature Tracking and Facial Expression Recognition[J]. IEEE Transactions on Image Processing, 2013, 22(7):2559-2573. doi: 10.1109/TIP.2013.2253477
[20]	LOPESA T, AGUIAR E D, SOUZA A F D, et al. Facial Expression Recognition with Convolutional Neural Networks:Coping with Few Data and The Training Sample Order[J]. Pattern Recognition, 2017, 61:610-628. doi: 10.1016/j.patcog.2016.07.026
[21]	李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别[J]. 自动化学报, 2018, 44(1):176-182.
	LI Yong, LIN Xiaozhu, JIANG Mengying. Facial Expression Recognition with Cross-Connect LeNet-5 Network[J]. Acta Automatica Sinica, 2018, 44(1):176-182.
[22]	宋剑桥, 王峰, 牛锦, 等. 一种面向时空神经网络的潜在情绪识别方法[J]. 西安电子科技大学学报, 2021, 48(4):159-167.
	SONG Jianqiao, WANG Feng, NIU Jin, et al. Potential Emotion Recognition Based on the Fusion of the Spatio-Temporal Neural Network and Facial Pulse Signals[J]. Journal of Xidian University, 2021, 48(4):159-167.

AU	描述	样图
1	内眉抬起
2	外眉抬起
4	眉毛压低
5	上眼睑抬起
6	脸颊抬起
7	眼皮拉紧
9	鼻子皱起
10	上嘴唇抬起
12	嘴角拉紧
15	嘴角下拉
17	下巴抬起
20	嘴唇拉长
23	嘴唇紧闭
24	嘴唇下拉
26	下颚下拉

表情	相应AUs
愤怒	AU4,AU5,AU7,AU23,AU24
厌恶	AU9,AU17
恐惧	AU4,AU1+AU5,AU5+AU7
高兴	AU6,AU12,AU25
悲伤	AU1,AU4,AU15,AU17
惊讶	AU5,AU26,AU27,AU1+AU2

AU局部候选区域	AU编号	特征区域	产生表情
AUg₁	AU1,AU2,AU5,AU7	1,2,5,6,8,9,12,13,40,41,42,43	愤怒、恐惧、悲伤、惊讶
AUg₂	AU4	1,2,3,4,5,6,8,9,12,13,40,41	愤怒、恐惧、悲伤
AUg₃	AU6	16,17,18,19,42,43	高兴
AUg₄	AU9	10,11,17,18,22,23	厌恶
AUg₅	AU12,AU15	21,22,23,24,25,26,27,28,37	高兴、悲伤、惊讶
AUg₆	AU25,AU26,AU27	25,26,27,28,29,30,31,32,33,34,35,36,37	高兴、惊讶
AUg₇	AU17	29,30,31,32,33,34,35,36	厌恶、悲伤
AUg₈	AU23,AU24	26,27,29,30,31,32,37	愤怒

AU组名	特征区域1	特征区域2
AUg₁	1,2,5,6,8,9,12,13,40,41	7,8,9,12,13,14,40,41,42,43
AUg₂	8,9,10,11,12,13,40,41,42,43	1,2,3,4,5,6,10,11
AUg₃	16,17,18,19	16,17,18,19,22,23
AUg₄	17,18,22,23	10,11,17,18,19,22,23
AUg₅	22,23,26,27,37	15,20,21,22,23,24
AUg₆	25,26,27,28,37	25,26,27,28,37
AUg₇	29,30,31,32,33,34,35	29,30,31,32,33,34,35,36
AUg₈	26,27,37	22,23,26,27,37

表情	中性	愤怒	厌恶	恐惧	高兴	悲伤	惊讶
中性	100.00	0	0	0	0	0	0
愤怒	0	100.00	0	0	0	0	0
厌恶	0	0	100.00	0	0	0	0
恐惧	0	0	0	100.00	0	0	0
高兴	0	0	0	0	100.00	0	0
悲伤	0	0	0	0	0	100.00	0
惊讶	0	0	0	0	0	0	100.00

Facial expression recognition based on local representation

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 18

References 22

Related Articles 15

Metrics

Comments

Recommended 0

算法	中性	愤怒	厌恶	恐惧	高兴	悲伤	惊讶	平均	CPU每帧耗时/ms
S-Patches^[16]	83.29	88.74	99.12	98.43	89.97	90.64	95.22	92.20	5.86
M-scale^[17]	90.80	90.37	92.45	97.37	95.63	92.10	97.54	93.75	6.43
CNN^[20]	98.79	92.15	94.28	97.12	100.00	98.54	93.21	96.30	1 351.01
STNN^[22]	79.24	85.12	90.87	89.20	70.56	78.34	80.51	81.98	1.23
EAU-CNN	98.78	98.37	98.74	97.69	100.00	97.47	100	98.29	50.25

[1]	QU Jiahui, HE Jie, DONG Wenqian, LI Yunsong, ZHANG Tongzhen, YANG Yufei. Change detection method based on multi-scale and multi-resolution information fusion [J]. Journal of Xidian University, 2025, 52(1): 105-116.
[2]	TANG Shuyuan, ZHOU Yiqing, LI Jintao, LIU Chang, SHI Jinglin. Dual attention pedestrian detector for occlusion scenario based on feature calibration [J]. Journal of Xidian University, 2024, 51(6): 25-39.
[3]	HENG Hongjun, YU Longwei. Time series anomaly detection based on multi-scale feature information fusion [J]. Journal of Xidian University, 2024, 51(3): 203-214.
[4]	GONG Junyang, FU Weihong, FANG Houzhang. Research on lightweight and feature enhancement of SAR image ship targets detection [J]. Journal of Xidian University, 2024, 51(2): 96-106.
[5]	ZHONG Hao, BIAN Shan, WANG Chuntao. Real world image tampering localization combining the self-attention mechanism and convolutional neural networks [J]. Journal of Xidian University, 2024, 51(1): 135-146.
[6]	ZHANG Xinyu, LIANG Yu, ZHANG Wei. Real-time smoke segmentation algorithm combining global and local information [J]. Journal of Xidian University, 2024, 51(1): 147-156.
[7]	HUANG Heyuan, MU Caihong, FANG Yunfei, LIU Yi. Graph convolution neural network for recommendation using graph negative sampling [J]. Journal of Xidian University, 2024, 51(1): 86-99.
[8]	DU Mingyang, DU Meng, PAN Jifei, BI Daping. Generative adversarial model for radar intra-pulse signal denoising and recognition [J]. Journal of Xidian University, 2023, 50(6): 133-147.
[9]	FAN Wentong,LI Zhenyu,ZHANG Tao,LUO Xiangyang. JPEG image steganalysis based on deep extraction of stego noise [J]. Journal of Xidian University, 2023, 50(4): 157-169.
[10]	WANG Kan, WANG Mengyang, LIU Xin, TIAN Guoqiang, LI Chuan, LIU Wei. Event detection by combining self-attention and CNN-BiGRU [J]. Journal of Xidian University, 2022, 49(5): 181-188.
[11]	SHI Yunlong,YUAN Wenhao,HU Shaodong,LOU Yingxi. Convolutional quasi-recurrent network for real-time speech enhancement [J]. Journal of Xidian University, 2022, 49(3): 183-190.
[12]	ZHOU Peng,YANG Jun. Index edge geometric convolution neural network for point cloud classification [J]. Journal of Xidian University, 2022, 49(2): 207-217.
[13]	YANG Jun,WANG Xingxing,LU Youpeng. Shape correspondence calculation using the unsupervised siamese functional maps network [J]. Journal of Xidian University, 2022, 49(1): 225-235.
[14]	YU Haoyang,YIN Liang,LI Shufang,LV Shun. Recognition algorithm for the little sample radar modulation signal based on the generative adversarial network [J]. Journal of Xidian University, 2021, 48(6): 96-104.
[15]	SONG Jianfeng,MIAO Qiguang,WANG Chongxiao,XU Hao,YANG Jin. Multi-scale single object tracking based on the attention mechanism [J]. Journal of Xidian University, 2021, 48(5): 110-116.