用于面瘫分级的自监督非对称特征学习方法

doi:10.19665/j.issn1001-2400.2021.03.015

摘要/Abstract

摘要：

面瘫是一种以面部表情肌群运动功能障碍为主要特征的疾病。利用人工智能技术进行面瘫分级的辅助诊断不仅能够提高诊断的效率,且能够降低诊断结果受主观医疗经验的影响,提高诊断的准确性。而现有的基于计算机视觉的面瘫辅助诊断评估方法或模型主要存在以下3个问题:① 从静态面部图像中提取的面部非对称特征难以准确表达面部运动的非对称特征;② 浅层机器学习模型难以准确提取有效的面部特征信息;③ 深度神经网络模型难以从小规模面瘫视频数据中学习有效面部运动差异特征。为解决这些问题,提出一种用于面瘫分级评估的自监督面部非对称特征学习方法。该方法首先利用视频序列预测任务作为面瘫分级评估的上游任务,可充分地利用大量无标签数据对基于三维卷积神经网络的深度神经网络模型(C3D,R3D和R(2+1)D)进行预训练,提升模型对患者面部运动特征的学习能力;然后,将预训练后的模型迁移至面瘫分级评估的下游任务,提升面瘫分级评估的准确性。此外,由于面瘫分级评估主要依据患者整个面部和面部局部区域的运动非对称性,因此提出的模型结合了面部整体和局部区域的时序特征,并通过计算运动的非对称性程度来进行最终的面瘫分级评估。实验结果证明,该模型相对于目前现有的面瘫分级方法,在准确性、召回率和F1等指标上均有较大程度的提升。

关键词: 面瘫分级, 自监督学习, 辅助诊断, 非对称性运动, 深度学习

Abstract:

Facial paralysis is a kind of facial disease with the functional disorder of the facial expression muscle,and adversely affects the patients’ mental and physical health.The aided diagnosis and evaluation system of facial paralysis based on computer vision can reduce the influence of doctors’ subjective experience on the diagnosis results,and improve the accuracy and efficiency of diagnosis.However,the existing evaluation methods suffer mainly from three drawbacks:① the facial asymmetry features are extracted from the static facial images,which can not represent the asymmetrical features of facial movements;② the shallow machine learning models have their limitations on extracting useful facial features;③ it is difficult for depth models to learn the effective facial features from small-scaled videos of facial paralysis.To solve these problems,we present an automatic facial paralysis evaluation method based on self-supervised facial asymmetry learning (self-SFAL).The key idea behind our method is that the pre-trained 3D-CNNs on the pretext task are transferred to extract the patients’ facial spatiotemporal features for the downstream task of facial paralysis evaluation.With the assistance of the pretext task,the 3D-CNNs can leverage numerous videos without any labels,and can be easily pre-trained to adapt to our facial paralysis evaluation task.Furthermore,inspired by the doctor’s evaluating facial paralysis by focusing on the asymmetry of the facial muscle movements on both the patients’ whole faces and the involved facial regions,our method combines the global and local facial spatiotemporal features for the final facial paralysis evaluation.Experimental results have verified a better performance of the proposed method,with accuracy,Recall and F1 improved.

Key words: facial nerve paralysis, self-supervision, auxiliary diagnosis, non-symmetrical movement, deep learning

中图分类号:

TG156

孙豪杰,李苗钰,章盼盼,许鹏飞. 用于面瘫分级的自监督非对称特征学习方法[J]. 西安电子科技大学学报, 2021, 48(3): 115-122.

SUN Haojie,LI Miaoyu,ZHANG Panpan,XU Pengfei. Self-supervised facial asymmetry learning for automatic evaluation of facial paralysis[J]. Journal of Xidian University, 2021, 48(3): 115-122.

图/表 5

图1

图2

图3

图4

表1

参考文献 28

[1]	WANG T, ZHANG S, DONG J, et al. Automatic Evaluation of the Degree of Facial Nerve Paralysis[J]. Multimedia Tools and Applications, 2016,75(19):11893-11908. doi: 10.1007/s11042-015-2696-0
[2]	林杨. 面瘫患者面部运动功能自动分级方法研究[D]. 山东:中国海洋大学, 2008.
[3]	SONG A P, XU G L, DING X H, et al. Assessment for Facial Nerve Paralysis Based on Facial Asymmetry[J]. Australasian Physical and Engineering Sciences in Medicine, 2017,40(4):851-860. doi: 10.1007/s13246-017-0597-4
[4]	HE S, SORAGHAN J J, O'REILLY B F, et al. Quantitative Analysis of Facial Paralysis Using Local Binary Patterns in Biomedical Videos[J]. IEEE Transactions on Biomedical Engineering, 2009,56(7):1864-1870. doi: 10.1109/TBME.2009.2017508
[5]	GUO Z, SHEN M, DUAN L, et al. Deep Assessment Process:Objective Assessment Process for Unilateral Peripheral Facial Paralysis Via Deep Convolutional Neural Network[C]// Proceedings of the 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).Piscataway:IEEE, 2017: 135-138.
[6]	YOSHIHARA H, SEO M, NGO T H, et al. Automatic Feature Point Detection Using Deep Convolutional Networks for Quantitative Evaluation of Facial Paralysis[C]// Proceedings of the 2016 9th International Congress on Image and Signal Processing,BioMedical Engineering and Informatics(CISP-BMEI).Piscataway:IEEE, 2016: 811-814.
[7]	陈莉明, 邓德祥. 混合深度卷积神经网络对人脸年龄的分类[J]. 华中科技大学学报(自然科学版), 2019,47(03):104-108.
	CHEN L, DENG D. Face Age Classification Based on Hybrid Deep Convolution Neural Network[J]. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2019,47(03):104-108.
[8]	ALLEN Z Z, LI Y Z. What Can ResNet Learn Efficiently,Going Beyond Kernels?[C] Proceedings of the NeurIPS:Neural Information Processing Systems.Piscataway:IEEE, 2019: 9017-9028.
[9]	徐彬, 陈渤, 刘家麒, 等 .采用双向 LSTM 模型的雷达 HRRP 目标识别[J]. 西安电子科技大学学报, 2019,46(2):29-34.
	XU Bin, CHEN Bo, LIU Jiaqi, et al. Radar HRRP Target Recognition by the Bidirectional LSTM Model[J]. Journal of Xidian University, 2019,46(2):29-34.
[10]	XU P, WANG L, GUAN Z, et al. Evaluating Brush Movements for Chinese Calligraphy:a Computer Vision Based Approach[C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence.Piscataway:IEEE, 2018: 1050-1056.
[11]	YAO G, LEI T, ZHONG J. A Review of Convolutional-Neural-Network-Based Action Recognition[J]. Pattern Recognition Letters, 2019,118:14-22. doi: 10.1016/j.patrec.2018.05.018
[12]	EUSEBIO J M A. Convolutional Neural Networks for Facial Expression Recognition[C/OL]. [2020-12-12]. https://repository.asu.edu/items/37376.
[13]	REDDY S P T, KARRI S T, DUBEY S R, et al. Spontaneous Facial Micro-Expression Recognition Using 3D Spatiotemporal Convolutional Neural Networks[C]// Proceedings of the 2019 International Joint Conference on Neural Networks(IJCNN).Piscataway:IEEE, 2019: 1-8.
[14]	梁晓曦, 蔡晓东, 王萌, 等. NRCNN 与角度度量融合的人脸识别方法[J]. 西安电子科技大学学报, 2018,45(6):144-149.
	LIANG Xiaoxi, CAI Xiaodong, WANG Meng, et al. Face Recognition Method for Integrating the Nested Residual CNN and Angular Metric[J]. Journal of Xidian University, 2018,45(6):144-149.
[15]	JING L, TIAN Y. Self-Supervised Visual Feature Learning with Deep Neural Networks:a Survey[J/OL]. [2020-12-12]. http://ieeexplore.ieee.org/document/9086055.doi:101109/TPAMI.2020.2992393.
[16]	SAMSUDIN W S W, SUNDARAJ K. Clinical and Non-Clinical Initial Assessment of Facial Nerve Paralysis:a Qualitative Review[J]. Biocybernetics and Biomedical Engineering, 2014,34(2):71-78. doi: 10.1016/j.bbe.2014.02.005
[17]	BANSAL M, SHAH A, GOSAI B, et al. A Simple,Objective,and Mathematical Grading Scale for the Assessment of Facial Nerve Palsy[J]. Otology and Neurotology, 2020,41(1):105-114. doi: 10.1097/MAO.0000000000002450
[18]	LIU L, CHENG G, DONG J, et al. Evaluation of Facial Paralysis Degree Based on Regions[C]// Proceedings of the 2010 Third International Conference on Knowledge Discovery and Data Mining.Piscataway:IEEE, 2010: 514-517.
[19]	MODERSOHN L, DENZLER J. Facial Paresis Index Prediction by Exploiting Active Appearance Models for Compact Discriminative Features[C]// Proceedings of the International Conference on Computer Vision Theory and Applications.Piscataway:IEEE, 2016: 271-278.
[20]	MA L. Research into Assessment of Facial Paralysis Based on Key Points and Regions[D]. Shandong:Chinese Marine University, 2009.
[21]	NGO T H, SEO M, CHEN Y W, et al. Quantitative Assessment of Facial Paralysis Using Local Binary Patterns and Gabor Filters[C]// Proceedings of the Fifth Symposium on Information and Communication Technology.Piscataway:IEEE, 2014: 155-161.
[22]	NGO T H, SEO M, MATSUSHIRO N, et al. Quantitative Analysis of Facial Paralysis Based on Filters of Concentric Modulation[C]// Proceedings of the 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery(FSKD).Piscataway:IEEE, 2015: 1758-1763.
[23]	NISHIDA T, CHEN Y W, MATSUSHIRO N, et al. An Image Based Quantitative Evaluation Method for Facial Paralysis[C]// Proceedings of the 2nd International Conference on Software Engineering and Data Mining.Piscataway:IEEE, 2010: 706-709.
[24]	BARBOSA J, LEE K, LEE S, et al. Efficient Quantitative Assessment of Facial Paralysis Using Iris Segmentation and Active Contour-Based Key Points Detection with Hybrid Classifier[J]. BMC Medical Imaging, 2016,16(1):23. doi: 10.1186/s12880-016-0117-0
[25]	LIU X, DONG S, AN M, et al. Quantitative Assessment of Facial Paralysis Using Infrared Thermal Imaging[C]// Proceedings of the 2015 8th International Conference on Biomedical Engineering and Informatics (BMEI).Piscataway:IEEE, 2015: 106-110.
[26]	WANG X, HE K, GUPTA A. Transitive Invariance for Self-Supervised Visual Representation Learning[C]// Proceedings of the 2017 IEEE International Conference on Computer Vision.Piscataway:IEEE, 2017: 1338-1347.
[27]	XU D, XIAO J, ZHAO Z, et al. Self-Supervised Spatiotemporal Learning Via Video Clip Order Prediction[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019: 10334-10343.
[28]	GIRSHICK R. Fast R-CNN[C]// Proceedings of the 2015 IEEE International Conference on Computer Vision.Piscataway:IEEE, 2015: 1440-1448.

方法	评估指标
方法	准确率	平均精度	召回率	F1
Gabor+SVM	0.608 7	0.673 1	0.544 6	0.580 1
LBP+SVM	0.648 8	0.567 3	0.542 3	0.501 8
GoogleNet	0.567 5	0.595 8	0.601 4	0.593 6
VGG-16	0.638 9	0.658 0	0.629 2	0.634 6
Resnet 34	0.628 6	0.672 6	0.687 6	0.641 9
Resnet 50	0.666 7	0.686 2	0.681 4	0.679 0
Resnet 101	0.657 1	0.727 9	0.630 6	0.654 4
CNN-FER	0.388 9	0.578 6	0.388 8	0.4589 5
MicroExpSTCNN	0.510 2	0.530 7	0.510 2	0.516 5
LSTM	0.653 1	0.690 0	0.704 1	0.697 0
CNN-LSTM	0.636 4	0.670 5	0.641 7	0.647 5
C3D	0.842 8	0.861 3	0.842 1	0.847 0
C3D(自监督)	0.890 8	0.898 5	0.899 5	0.896 6
R3D	0.746 7	0.780 9	0.770 3	0.760 2
R3D(自监督)	0.903 9	0.910 9	0.910 6	0.909 6
R(2+1)D	0.816 6	0.808 0	0.811 7	0.825 8
R(2+1)D(自监督)	0.851 5	0.843 4	0.853 1	0.847 8