利用多头-连体神经网络实现障碍行为识别

doi:10.19665/j.issn1001-2400.2022.04.012

摘要/Abstract

摘要：

障碍行为指有特殊需要人群的伤害性行为,对其进行分类和识别是人类行为识别的一个重要分支。针对多传感器设备布设于人体不同部位来感知障碍行为的过程中存在传感器之间相关性未能被考量的问题,基于深度学习理论,提出了利用多头-连体神经网络表征传感器之间相关性的方法。该网络在权值共享的基础上,构建多个子网络对多传感器数据分别进行一致的特征提取操作,并对所提取的特征进行融合后通过分类器进行识别。首先对于传感器中存在的缺失数据进行上采样并进行数据标准化分析,其次以贝叶斯优化对该网络的超参数进行选取。然后,由于引入Adam优化器在障碍行为识别领域存在着过拟合的问题,改用AdamW优化器进行L2正则化处理并能够提升网络的识别准确率。实验结果表明:此网络对障碍行为的分类准确率约达到96.0%,相对于基线网络和单头网络,分类准确率分别提高了约6.1%和8.8%并降低了部分行为之间存在的误识问题。相对于多头网络分类准确率提高了约2.4%,并减少了约92.22%的训练参数量。通过解决传感器之间的关联性问题,证明此网络对于障碍行为的识别具有有效性。

关键词: 行为识别, 深度学习, 神经网络, 权值共享, 特征提取

Abstract:

Impaired behavior recognition is an important branch of human activity recognition,which refers to harmful behavior of people with special needs.Aiming at the problem that the correlation between sensors is not taken into account when recognizing the impaired behavior by using multi-sensor devices equipped on different parts of the human body,based on deep learning theory,this paper proposes a multi-head-siamese neural network to characterize the relation between sensors,which builds multiple sub-networks for consistent feature extraction.The extracted features are fused and recognized by the classifier on the basis of the weight sharing idea.In the presented network,the upsampling operation is first employed to fill the missing collected data,and the data is then standardized to improve the recognition accuracy.Besides,the network hyperparameters are adjusted by the Bayesian optimization.In addition,due to the over-fitting problem when recognizing impaired behavior by introducing the Adam optimizer,L2 regularization is performed by using the AdamW optimizer,thus further improving the recognition accuracy.Processing results of raw data show that the network achieves a classification accuracy of 96.0%.Compared with the baseline network and single input network,the accuracy of the proposed network increases by 6.1% and 8.8%,respectively,and it could reduce the possibility of incorrect prediction.Compared with the multiple input network,its accuracy increases by 2.4%,and it reduces the number of training parameters by 92%.It is proved that this network is effective for impaired behavior recognition in terms of utilizing the relationship between sensors.

Key words: behavior recognition, deep learning, neural network, weight sharing, feature extraction

中图分类号:

TP391.4

马仑,刘鑫,赵斌,王瑞平,廖桂生,张亚静. 利用多头-连体神经网络实现障碍行为识别[J]. 西安电子科技大学学报, 2022, 49(4): 100-108.

MA Lun,LIU Xin,ZHAO Bin,WANG Ruiping,LIAO Guisheng,ZHANG Yajing. Impaired behavior recognition by using the multi-head-siamese neural network[J]. Journal of Xidian University, 2022, 49(4): 100-108.

图/表 12

图1

表1

图2

图3

图4

表2

图5

图6

表3

图7

图8

表4

参考文献 18

[1]	DEMROZI F, PRAVADELLI G, BIHORAC A, et al. Human Activity Recognition Using Inertial,Physiological and Environmental Sensors:a Comprehensive Survey[J]. IEEE Access, 2020, 8:210816-210836. doi: 10.1109/ACCESS.2020.3037715
[2]	DANGL M, MIN K, WANG H, et al. Sensor-Based and Vision-Based Human Activity Recognition:A Comprehensive Survey[J]. Pattern Recognition, 2020, 108:107561. doi: 10.1016/j.patcog.2020.107561
[3]	XIA K, HUANG J, WANG H. LSTM-CNN Architecture for Human Activity Recognition[J]. IEEE Access, 2020, 8:56855-56866. doi: 10.1109/ACCESS.2020.2982225
[4]	IHIANLEI K, NWAJANA A O, EBENUWA S H, et al. A Deep Learning Approach for Human Activities Recognition from Multimodal Sensing Devices[J]. IEEE Access, 2020, 8:179028-179038. doi: 10.1109/ACCESS.2020.3027979
[5]	SHENG T, HUBER M. Siamese Networks for Weakly Supervised Human Activity Recognition[C]// 2019 IEEE International Conference on Systems,Man and Cybernetics.Piscataway:IEEE, 2019:4069-4075.
[6]	BROMLEY J, GUYON I, LECUN Y, et al. Signature Verification Using a "Siamese" Time Delay Neural Network[J]. International Journal of Pattern Recognition and Artificial Intelligence, 1993, 7(4):669-688. doi: 10.1142/S0218001493000339
[7]	郭惠勇. 多传感器信息融合技术的研究与进展[J]. 中国科学基金, 2005, 1:5.
	GUO Hui Yong. Researches and Advances in Multi-Sensor Information Fusion Technology[J]. Bulletin of National Natural Science Foundation of China, 2005, 1:5.
[8]	余志超, 张瑞红. 结合深度轮廓特征的改进孪生网络跟踪算法[J]. 西安电子科技大学学报, 2020, 47(3):40-49.
	YU Zhichao, ZHANG Ruihong. Improved Siamese Network Based Object Tracking Combined with the Deep Contour Feature[J]. Journal of Xidian University, 2020, 47(3):40-49.
[9]	HAMMERLA N Y, HALLORAN S, PLÖTZ T. Deep,Convolutional,and Recurrent Models for Human Activity Recognition Using Wearables[C]// Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. New York: AAAI, 2016:1533-1540.
[10]	SNOEK J, LAROCHELLE H, ADAMS R P. Practical Bayesian Optimization of Machine Learning Algorithms[EB/OL]. [2012-06-13]. https://arxiv.org/abs/1206.2944.
[11]	LOSHCHILOV I, HUTTER F. Decoupled Weight Decay Regularization[EB/OL]. [2017-11-14]. https://doi.org/10.48550/arXiv.1711.05101.
[12]	BAO L, INTILLE S S. Activity Recognition from User-Annotated Acceleration Data[C]// International Conference on Pervasive Computing.Berlin:Springer, 2004:1-17.
[13]	HSU Y L, YANG S C, CHANG H C, et al. Human Daily and Sport Activity Recognition Using a Wearable Inertial Sensor Network[J]. IEEE Access, 2018:31715-31728.
[14]	LIU R, PENG L, TONG L, et al. The Design of Wearable Wireless Inertial Measurement Unit for Body motion Capture System[C]// 2018 IEEE International Conference on Intelligence and Safety for Robotics.Piscataway:IEEE, 2018:557-562.
[15]	SI C, JING Y, WANG W, et al. Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning[C]// Proceedings of the European Conference on Computer Vision.Berlin:Springer, 2018:103-118.
[16]	王波, 邓科. DB-SMOTE及多层堆叠用于心律失常识别[J]. 西安电子科技大学学报, 2021, 48(4):136-143.
	WANG Bo, DENG Ke. DB-SMOTE and Multi-Layer Stacking Used for Arrhythmia Recognition[J]. Journal of Xidian University, 2021, 48(4):136-143.
[17]	HAN H, WANG W Y, MAO B H. Borderline-SMOTE:A New Over-Sampling Method in Imbalanced Data Sets Learning[C]// International Conference on Intelligent Computing.Berlin:Springer, 2005:878-887.
[18]	AHMAD W, KAZMI B M, ALI H. Human Activity Recognition Using Multi-Head CNN Followed by LSTM[C]// 2019 15th International Conference on Emerging Technologies.Piscataway:IEEE, 2019:1-6.

超参数	取值	超参数	取值
学习率	0.008	最大池化层大小	4
卷积核数目1	80	权重衰减	0.000 006
卷积核数目2	96	Dropout	0.4
卷积核数目3	112	Batchsize	240
卷积核长度	4	Epochs	80

状态	手腕	脚踝	胸部
走路	21 400	21 600	21 400
打头	21 600	21 600	21 600
撞墙	21 600	21 400	21 600
打人	21 600	21 600	21 600
抽搐	21 600	21 400	21 600
缺失比例	0.186	0.372	0.186

基线网络	文献[3]的网络	文献[4]的网络			文中提出的网络
Input Layer:0	Input Layer:0	(处理手部传	(处理腿踝	(处理胸部	Input Layer(hand):0
cnn1:8 720	lstm1:7 680	感器数据)	传感器数据)	传感器数据)	Input Layer(leg):0
max_pooling1:0	lstm2:8 320	Input Layer:0	Input Layer:0	Input Layer:0	Input Layer(neck):0
cnn2:30 816	cnn1:10 304	cnn11:1 792	cnn21:2 944	cnn31:6 400	cnn1:2 240
max_pooling2:0	max_pooling:0	max_pooling11:0	max_pooling21:0	max_pooling31:0	max_pooling1:0
cnn3:43 120	cnn2:24 704	cnn12:12 352	cnn22:20 544	cnn32:45 120	cnn2:23 136
GAP:0	GAP:0	max_pooling12:0	max_pooling22:0	max_pooling32:0	max_pooling2:0
dropout:0	BN:512	cnn13:12 352	cnn23:20 544	cnn33:45 120	cnn3:32 368
Dense:565	Dense:645	max_pooling13:0	max_pooling23:0	max_pooling33:0	GAP:0
		Bi_lstm1:197 632	Bi_lstm2:197 632	Bi_lstm3:197 632	dropout:0
		Dropout1:0	Dropout2:0	Dropout3:0	concatenate:0
					Dense:1 685
		Concatenate:0
		Dense:3 845
训练参数量: 83 221	训练参数量: 51 909	训练参数量: 763 909			训练参数量: 59 429

行为	基线网络	文献[3]的模型	文献[4]的模型	文中的模型
走路f1-score	0.774	0.812	0.935	0.929
打头f1-score	0.829	0.753	0.908	0.937
撞墙f1-score	0.958	0.934	0.963	0.991
打别人f1-score	0.970	0.921	0.966	0.983
抽搐f1-score	0.955	0.939	0.915	0.965
准确率	0.899	0.872	0.936	0.960