基于时序行为检测的工作流识别

doi:10.16180/j.cnki.issn1007-7820.2021.08.003

Abstract

Abstract:

In a complex manufacturing environment, it is difficult for workflow recognition based on action recognition to localize the start and end times of each activity in a workflow from the video. In order to localize the activities temporally in the workflow from the video, according to R-C3D, a workflow recognition framework based on temporal action detection is proposed. The proposed workflow recognition method introduces a random sparse sampling strategy to reduce the redundancy between adjacent frames, and uses Res3D network to extract spatio-temporal features in the video. In addition, Soft-NMS strategy is used to eliminate the highly overlapping and low confidence proposals. Experiment results show that compared with R-C3D, the proposed method can improve the speed of training and detecting about 80% and 85%, respectively without loss of accuracy, proving the effectiveness of the method for workflow identification.

Key words: intelligent manufacturing, workflow recognition, action recognition, temporal action detection, temporal localization, sparse sampling, spatio-temporal features, Soft-NMS

CLC Number:

TP312

WANG Qingwen,HU Haiyang. Workflow Recognition Based on Temporal Action Detection[J].Electronic Science and Technology, 2021, 34(8): 14-18.

Figures/Tables 4

References 17

[1]	Jin Y, Dou Q, Chen H, et al. SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network[J]. IEEE Transactions on Medical Imaging, 2017,37(5):1114-1126.
[2]	Yang H, Park M, Cho M, et al. A system architecture for manufacturing process analysis based on big data and process mining techniques[C]. Santa Clara:IEEE International Conference on Big Data, 2015.
[3]	袁小平, 王岗, 王晔枫, 等. 基于改进卷积神经网络的交通标志识别方法[J]. 电子科技, 2019,32(11):28-32.
	Yuan Xiaoping, Wang Gang, Wang Yefeng, et al. Traffic sign recognition method based on improved convolutional neural network[J]. Electronic Science and Technology, 2019,32(11):28-32.
[4]	周文凯, 韩芳, 孔维健. 基于Faster-RCNN的极验点选式验证码识别[J]. 电子科技, 2019,32(9):42-45,59.
	Zhou Wenkai, Han Fang, Kong Weijian. Point-selective geetest CAPTCHA recognition based on Faster-RCNN[J]. Electronic Science and Technology, 2019,32(9):42-45,59.
[5]	胡海洋, 丁佳民, 胡华, 等. 基于三维卷积神经网络的工作流识别方法[J]. 计算机集成制造系统, 2018,24(7):1747-1757.
	Hu Haiyang, Ding Jiamin, Hu Hua, et al. Workflow recognition method based on 3D convolutional neural networks[J]. Computer Integrated Manufacturing Systems, 2018,24(7):1747-1757.
[6]	Slama R, Wannous H, Daoudi M, et al. Accurate 3D action recognition using learning on the Grassmann manifold[J]. Pattern Recognition, 2015,48(2):556-567.
[7]	Xu H, Das A, Saenko K. R-C3D: Region convolutional 3d network for temporal activity detection[C]. Venice: The IEEE International Conference on Computer Vision, 2017.
[8]	Hu H, Cheng K, Li Z, et al. Workflow recognition with structured two-stream convolutional networks[J]. Pattern Recognition Letters, 2020,130(2):267-274.
[9]	Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos[C]. Montreal: Advances in Neural Information Processing Systems, 2014.
[10]	Bodla N, Singh B, Chellappa R, et al. Soft-NMS--improving object detection with one line of code[C]. Venice: The IEEE International Conference on Computer Vision, 2017.
[11]	Wang L, Xiong Y, Zhe W, et al. Temporal segment networks: towards good practices for deep action recognition[C]. Amsterdam:European Conference on Computer Vision, 2016.
[12]	Hara K, Kataoka H, Satoh Y, et al. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet[C]. Salt Lake City:The IEEE Computer Vision and Pattern Recognition, 2018.
[13]	Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3D convolutional networks[C]. Santiago: IEEE International Conference on Computer Vision, 2015.
[14]	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]. Nevada:The IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[15]	Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017,39(6):1137-1149.
[16]	Neubeck A, Gool L. Efficient non-maximum suppression[C]. Hong Kong:The Eighteenth International Conference on Pattern Recognition, 2006.
[17]	董海青, 林宝军, 刘迎春. 一种有效掏运动目标周围“空洞”的算法[J]. 电子设计工程, 2018,26(18):25-29.
	Dong Haiqing, Lin Baojun, Liu Yingchun. An algorithm to effectively suppress "hole"around moring objects[J]. Electronic Design Engineering, 2018,26(18):25-29.

算法	单轮训练时间	FPS	mAP@0.5/%
R-C3D	1.13 h	389	67.5
Proposed method	0.61 h	722	68.7

IoU AP/mAP	0.3	0.4	0.5	0.6
切割厚板	80.5	79.3	78.0	71.6
清理厚板	69.5	71.2	70.2	54.3
搬运厚板	81.0	80.4	79.5	69.2
厚板装车	49.9	47.2	46.9	45.1
mAP	70.2	69.8	68.7	60.1

Workflow Recognition Based on Temporal Action Detection

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 4

References 17

Related Articles 5

Metrics

Comments

Recommended 10

[1]	ZHANG Ying,LIU Zilong,WAN Wei. UAV Vehicle Target Detection Based on Faster R-CNN [J]. Electronic Science and Technology, 2021, 34(11): 11-20.
[2]	WU Qian,WU Fei,LUO Lizhi. Human Skeleton-based Action Recognition Based on Deep Learning [J]. Electronic Science and Technology, 2020, 33(11): 79-83.
[3]	GUO Huanyu,HOU Yuemin,LI Kang. RFID Positioning Method and Its Application in Intelligent Manufacturing [J]. , 2017, 30(4): 115-.
[4]	HAN Junhao,ZHAO Huaixun. An Improved Human Behavior Detection and Tracking Algorithm [J]. , 2014, 27(9): 25-.
[5]	HAN Junhao,ZHAO Huaixun. Human Behaviour Extraction Method Based on Shape Features and Movement Features [J]. , 2014, 27(10): 6-.