维度情感模型下的表情图像生成及应用

doi:10.19665/j.issn1001-2400.2021.05.005

摘要/Abstract

摘要：

为了解决基于深度学习的人脸表情识别所需训练数据包含表情类别有限且训练数据规模不均衡的问题,提出了Arousal-Valence维度情感空间中基于生成对抗网络的表情图像生成方法AV-GAN,用于生成更多样且均衡的表情识别训练数据。该方法使用标记分布表示表情图像,通过引入身份控制和表情控制模块,以及对抗学习方法实现在Arousal-Valence空间中随机采样和生成表情图像。在Oulu-CASIA数据库上的评估实验显示,使用本文方法对训练数据进行数据增强比使用原训练数据的表情识别准确率可提升6.5%,证明了该方法能有效地提升非均衡训练数据下的表情识别准确率。

关键词: Arousal-Valence情感模型, 生成对抗网络, 图像生成, 数据增强, 人脸表情识别

Abstract:

In order to solve the problem that the training data of deep learning based facial expression recognition methods usually cover a limited part of the expression space and have an imbalanced distribution,we propose AV-GAN,a facial expression image synthesis method in Arousal-Valence dimensional emotion space,based on the generative adversarial network,to generate more diverse and balanced facial expression training data.The method uses label distribution to represent the expression for the face image,and employs an identity control module,an expression control module,and adversarial learning to realize the random sampling and generation of expression images in Arousal-Valence space.Evaluations on Oulu-CASIA database show that the accuracy of the recognition of the facial expression using the proposed method to augment training data is increased by 6.5%,compared with that using the original training data.It is proved that the proposed method can effectively improve the facial expression recognition accuracy under imbalanced training data.

Key words: Arousal-Valence emotion model, generative adversarial network, image generation, data augmentation, facial expression recognition

中图分类号:

TP391

杨静波,赵启军,吕泽均. 维度情感模型下的表情图像生成及应用[J]. 西安电子科技大学学报, 2021, 48(5): 30-37.

YANG Jingbo,ZHAO Qijun,LYU Zejun. Synthesis of the expression image and its application under the dimentional emotion model[J]. Journal of Xidian University, 2021, 48(5): 30-37.

图/表 10

图1

图2

图3

图4

图5

图6

图7

表1

表2

表3

参考文献 20

[1]	LI S, DENG W H. Deep Facial Expression Recognition:A Survey[C]// IEEE Transactions on Affective Computing.Piscataway:IEEE, 2020:1-1.
[2]	EKMAN P, FRIESEN W V. Constants Across Cultures in the Face and Emotion[J]. Journal of Personality and Social Psychology, 1971, 17(2):124-129. doi: 10.1037/h0030377
[3]	杨勇, 黄文波, 金裕成, 等. 基于MSVR和Arousal-Valence情感模型的表情识别研究[J]. 重庆邮电大学学报(自然科学版), 2016, 28(6):836-843.
	YANG Yong, HUANG Wenbo, JIN Yucheng, et al. Facial Expression Recognition Method Based on MSVR and Arousal-Valence Emotion Model[J]. Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition), 2016, 28(6):836-843.
[4]	耿新, 徐宁. 标记分布学习与标记增强[J]. 中国科学:信息科学, 2018, 48(5):521-530.
	GENG Xin, XU Ning. Label Distribution Learning and Label Enhancement (in Chinese)[J]. Sci Sin Inform, 2018, 48(5):521-530.
[5]	MOLLAHOSSEINI A, HASSANI B, MAHOOR M H. AffectNet:A Database for Facial Expression,Valence,and Arousal Computing in the Wild[J]. IEEE Transactions on Affective Computing, 2019, 10(1):18-31. doi: 10.1109/TAFFC.2017.2740923
[6]	EKMAN, P, FRIESEN, W V. Facial Action Coding System (FACS):A Technique for the Measurement of Facial Actions[J]. Rivista Di Psichiatria, 1978, 47(2):126-38.
[7]	HE Z L, ZUO W M, KAN M, et al. AttGAN:Facial Attribute Editing by Only Changing What You Want[J]. IEEE Transactions on Image Processing, 2019, 28(11):5464-5478. doi: 10.1109/TIP.83
[8]	YANG H Y, ZHANG Z, YIN L J. Identity-Adaptive Facial Expression Recognition through Expression Regeneration Using Conditional Generative Adversarial Networks[C]// 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).Piscataway:IEEE, 2018:294-301.
[9]	WANG W X, SUN Q, FU Y W, et al. Comp-GAN:Compositional Generative Adversarial Network in Synthesizing and Recognizing Facial Expression[C]// Proceedings of the 27th ACM International Conference on Multimedia.Nice:ACM, 2019:211-219.
[10]	DING H, SRICHARAN K, CHELLAPPA R. ExprGAN:Facial Expression Editing with Controllable Expression Intensity[C]// The Thirty-Second AAAI Conference on Artificial Intelligence.Louisiana:AAAI, 2018:6781-6788.
[11]	LINDT A, BARROS P, SIQUEIRA H, et al. Facial Expression Editing with Continuous Emotion Labels[C]// 2019 14th IEEE International Conference on Automatic Face Gesture Recognition.Piscataway:IEEE, 2019:1-8.
[12]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: MIT Press, 2014:2627-2680.
[13]	莫建文, 徐凯亮, 林乐平, 等. 结合互信息最大化的文本到图像生成方法[J]. 西安电子科技大学学报, 2019, 46(5):180-188.
	MO Jianwen, XU Kailiang, LIN Leping, et al. Text-to-Image Generation Combined with Mutual Information Maximization[J]. Journal of Xidian Universyt, 2019, 46(5):180-188.
[14]	贾宇峰, 马力. 条件约束下的自我注意生成对抗网络[J]. 西安电子科技大学学报, 2019, 46(6):163-171.
	JIA Yufeng, MA Li. Self-Attention Generative Adversarial Network with the Conditional Constraint[J]. Journal of Xidian Universyt, 2019, 46(6):163-171.
[15]	PARKHI O M, VEDALDI A, ZISSERMAN A. Deep Face Recognition[C]//The British Machine Vision Conference (BMVC 2015).Swansea:BMVC, 2015: 41.1-41.12.
[16]	HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition[C]// The 29th IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:770-778.
[17]	ZHAO G Y, HUANG X H, TAINI M, et al. Facial Expression Recognition from Near-Infrared Videos[J]. Image and Vision Computing, 2011, 29(9):607-619. doi: 10.1016/j.imavis.2011.07.002
[18]	ZHANG K P, ZHANG Z P, LI Z F, et al. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks[J]. IEEE Signal Processing Letters, 2016, 23(10):1499-1503. doi: 10.1109/LSP.2016.2603342
[19]	SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J/OL]. [2020-07-10]. https://arxiv.org/abs/1409.1556.
[20]	KINGMA D P, BA J. Adam:A Method for Stochastic Optimization[C/OL]. [2020-07-10]. https://arxiv.org/abs/1412.6980.

表情类别	训练数据	测试数据
生气	1 267	443
厌恶	1 227	326
恐惧	1 357	359
高兴	1 368	343
悲伤	1 235	363
惊讶	1 259	362
中性	366	114
合计	8 079	2 300

表情类别	训练数据	测试数据
生气	2 500	443
厌恶	2 500	326
恐惧	2 500	359
高兴	2 500	343
悲伤	2 500	353
惊讶	2 500	362
中性	2 500	114
合计	2 500	2 300

表情类别	基准方法	传统方法(随机镜像)^[1]	传统方法(随机旋转)^[1]	传统方法(随机裁剪)^[1]	AV-GAN方法
生气	42.50±15.69	51.25±8.16	48.25±5.07	48.75±12.09	54.25±7.79
厌恶	41.00±4.30	38.75±5.21	42.00±7.65	47.50±9.09	56.00±10.56
恐惧	55.00±16.17	50.25±14.04	52.00±8.90	44.75±17.15	52.25±12.71
高兴	63.75±4.76	72.75±6.57	72.25±5.11	74.50±8.67	71.50±3.28
悲伤	51.75±6.87	51.25±12.34	46.00±7.58	51.50±7.53	49.25±8.04
惊讶	64.25±3.11	67.00±4.30	61.75±3.77	65.75±6.87	67.75±8.50
中性	3.50±2.29	2.00±1.58	1.00±0.00	3.75±1.09	29.50±7.43
合计	50.25±4.76	52.75±5.11	51.00±3.94	52.75±6.01	56.75±6.98