基于特征校准的双注意力遮挡行人检测器

doi:10.19665/j.issn1001-2400.20240909

西安电子科技大学学报 ›› 2024, Vol. 51 ›› Issue (6): 25-39.doi: 10.19665/j.issn1001-2400.20240909

基于特征校准的双注意力遮挡行人检测器

汤书苑¹^,²^,³^,⁴(), 周一青¹^,²^,³^,⁴(), 李锦涛¹^,²^,³^,⁴(), 刘畅¹^,²^,⁴(), 石晶林¹^,²^,³^,⁴()

1.中国科学院计算技术研究所处理器芯片全国重点实验室,北京 100190
2.中国科学院计算技术研究所无线通信技术研究中心,北京100190
3.中国科学院大学,北京 100049
4.移动计算与新型终端北京市重点实验室,北京 100190

收稿日期:2024-01-06 出版日期:2024-12-20 发布日期:2024-10-08
通讯作者: 周一青(1975—),女,研究员,E-mail:zhouyiqing@ict.ac.cn
作者简介:汤书苑(1985—),女,中国科学院大学博士研究生,E-mail:tangshuyuan20b@ict.ac.cn;
李锦涛(1962—),男,研究员,E-mail:jtli@ict.ac.cn;
刘畅(1986—),男,助理研究员,E-mail:liuchang@ict.ac.cn;
石晶林(1972—),男,研究员,E-mail:sjl@ict.ac.cn
基金资助:
国家自然科学基金(U21A20449);江苏省重点研发计划(BE2021013-2)

Dual attention pedestrian detector for occlusion scenario based on feature calibration

TANG Shuyuan¹^,²^,³^,⁴(), ZHOU Yiqing¹^,²^,³^,⁴(), LI Jintao¹^,²^,³^,⁴(), LIU Chang¹^,²^,⁴(), SHI Jinglin¹^,²^,³^,⁴()

1. State Key Laboratory of Processors,Institute of Computing Technology,CAS,Beijing 100190,China
2. Institute of Computing Technology,Chinese Academy of Sciences,Beijing 100190,China
3. University of the Chinese Academy of Sciences,Beijing 100049,China
4. Beijing Key Laboratory of Mobile Computing and Pervasive Device,Beijing 100190,China

Received:2024-01-06 Online:2024-12-20 Published:2024-10-08

摘要/Abstract

摘要：

基于计算机视觉的行人检测技术面临的主要挑战之一是遮挡问题,包括自然环境中物体对行人造成的类间遮挡以及行人与行人之间的类内遮挡。这些交织的遮挡模式限制了行人检测器的性能。为此,在Faster R-CNN标准行人检测框架的基础上,提出了一种基于特征校准的双注意力检测网络。该网络首先通过监督学习生成注意力掩码,用以表征图像中的行人空间特征;然后将掩码与主干特征融合,并结合通道注意力机制,校准行人区域。该方法能够增强行人的可见区域,同时减弱遮挡部分对分类和回归的干扰。此外,提出了一种基于遮挡率的非均匀采样策略,专门针对难例进行采样,帮助网络更有效地学习复杂遮挡模式。实验结果表明,与标准行人检测器相比,所提方法在CityPersons验证集的合理遮挡子集上性能提升了约2.5%。

关键词: 卷积神经网络, 行人检测, 双注意力机制, 特征校准, 难例挖掘, 遮挡率

Abstract:

One of the major challenges faced by pedestrian detection technology based on computer vision is the issue of occlusion,including inter-class occlusion caused by objects in the natural environment and intra-class occlusion between pedestrians.These intertwined occlusion patterns limit the performance of pedestrian detectors.To address this problem,this paper proposes a dual-attention detection network based on feature calibration within the standard Faster R-CNN pedestrian detection framework.The network first generates attention masks through supervised learning to represent the spatial features of pedestrians in the image.These masks are then fused with backbone features and combined with a channel attention mechanism to calibrate pedestrian regions.This approach enhances the visibility of pedestrian regions while reducing the impact of occluded parts on classification and regression.Additionally,a non-uniform sampling strategy based on occlusion rates is introduced,targeting hard examples to allow the network to better learn complex occlusion patterns.Experimental results demonstrate that in comparison with standard pedestrian detectors,the proposed method achieves a 2.5% performance improvement on the reasonable occlusion subset of the CityPersons validation dataset.

Key words: convolutional neural network, pedestrian detection, dual attention mechanism, feature calibration, hard example mining, occlusion rate

中图分类号:

TP391

汤书苑, 周一青, 李锦涛, 刘畅, 石晶林. 基于特征校准的双注意力遮挡行人检测器[J]. 西安电子科技大学学报, 2024, 51(6): 25-39.

TANG Shuyuan, ZHOU Yiqing, LI Jintao, LIU Chang, SHI Jinglin. Dual attention pedestrian detector for occlusion scenario based on feature calibration[J]. Journal of Xidian University, 2024, 51(6): 25-39.

图/表 14

图1

表1

图2

图3

图4

图5

表2

表3

表4

表5

图6

图7

表6

表7

参考文献 47

[1]	LIN C, LU J, WANG G, et al. Graininess-Aware Deep Feature Learning for Robust Pedestrian Detection[J]. IEEE Transactions Image Processing, 2020, 29:3820-3834.
[2]	LIU W, LIAO S, HU W. Efficient Single-Stage Pedestrian Detector by Asymptotic Localization Fitting and Multi-Scale Context Encoding[J]. IEEE Transactions Image Processing, 2020, 29:1413-1425.
[3]	CAO J, PANG Y, HAN J, et al. Taking a Look at Small-Scale Pedestrians and Occluded Pedestrians[J]. IEEE Transactions Image Processing, 2020, 29:3143-3152.
[4]	HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition(CVPR2016). Piscataway:IEEE, 2016:770-778.
[5]	REN S, HE K, GIRSHICK R, et al. Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(6):1137-1149.
[6]	王子晔, 苗夺谦, 赵才荣. 基于多粒度特征的行人跟踪检测结合算法[J]. 计算机研究与发展, 2020, 57(5):996-1002.
	WANG Zhiye, MIAO Duoqian, ZHAO Cairong, et al. A Pedestrian Tracking Algorithm Based on Multi-Granularity Feature[J]. Journal of Computer Research and Development, 2015, 57(5):996-1002.
[7]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal Loss for Dense Object Detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2):318-32.
[8]	REN J, CHEN X, LIU J, et al. Accurate Single Stage Detector Using Recurrent Rolling Convolution[C]// Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition(CVPR2017). Piscataway:IEEE, 2017:5420-5428.
[9]	SHAO Z, CHENG G, MA J, et al. Real-Time and Accurate UAV Pedestrian Detection for Social Distancing Monitoring in COVID-19 Pandemic[J]. IEEE Transactions on Multimedia, 2021, 24:2069-2083.
[10]	XIE J, PANG Y, KHAN M H, et al. Mask-Guided Attention Network and Occlusion-Sensitive Hard Example Mining for Occluded Pedestrian Detection[J]. IEEE Transactions on Image Processing, 2020, 30:3872-3884.
[11]	ZOU F, LI X, XU Q, et al. Correlation-and-Correction Fusion Attention Network for Occluded Pedestrian Detection[J]. IEEE Sensors Journal, 2023, 23(6):6061-6073.
[12]	XIE J, PANG Y, CHOLAKKAL H, et al. PSC-Net:Learning Part Spatial Co-ccurrence for Occluded Pedestrian Detection[J]. Science China Information Sciences, 2021, 64(2):1-13.
[13]	WU J, ZHOU C, ZHANG Q, et al. Self-Mimic Learning for Small-Scale Pedestrian Detection[C]// Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM, 2020:2012-2020.
[14]	WANG H, LI Y, WANG S. Fast Pedestrian Detection with Attention-Enhanced Multi-Scale RPN and Soft-Cascaded Decision Trees[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(12):5086-5093.
[15]	ZHANG T, YE Q, ZHANG B, et al. Feature Calibration Network for Occluded Pedestrian Detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 23(5):4151-4163.
[16]	XIE J, CHOLAKKAL H, ANWER R M, et al. Count-andSimilarity-Aware R-CNN for Pedestrian Detection[C]// Proceedings of the 16th European Conference on Computer Vision. Berlin:Springer, 2020:1-16.
[17]	WU J, ZHOU C, YANG M, et al. Temporal Context Enhanced Detection of Heavily Occluded Pedestrians[C]// Proceedings of the 33rd IEEE Conference on Computer Vision and Pattern Recognition(CVPR2020). Piscataway:IEEE, 2020:13430-13439.
[18]	NOH J, LEE S, KIM B, et al. Improving Occlusion and Hard Negative Handling for Single-Stage Pedestrian Detectors[C]// Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition(CVPR2018). Piscataway:IEEE, 2018:966-974.
[19]	ZHOU C, YUAN J. Multi-Label Learning of Part Detectors for Heavily Occluded Pedestrian Detection[C]// Proceedings of the 16th IEEE International Conference on Computer Vision(ICCV2017). Piscataway:IEEE, 2017:3486-3495.
[20]	ZHANG T, HAN Z, XU H, et al. CircleNet:Reciprocating Feature Adaptation for Robust Pedestrian Detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(11):4593-4604.
[21]	ZHANG S, YANG J, SCHIELE B. Occluded Pedestrian Detection Through Guided Attention in CNNs[C]// Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition(CVPR2018). Piscataway:IEEE, 2018:6995-7003.
[22]	ZHOU C, YUAN J. Bi-Box Regression for Pedestrian Detection and Occlusion Estimation[C]// Proceedings of the 15th European Conference on Computer Vision(ECCV2018). Berlin:Springer, 2018:135-151.
[23]	汪昱东, 郭继昌, 王天保. 一种改进的雾天图像行人和车辆检测算法[J]. 西安电子科技大学学报, 2020, 47(4):70-77.
	WANG Yudong, GUO Jichang, WANG Tianbao. Algorithm for Foggy-Image Pedestrian and Vehicle Detection[J]. Journal of Xidian University, 2020, 47(4):70-77.
[24]	LI W, HAN S, LIU Y. A Pedestrian Detection Algorithm Based on Channel Attention Mechanism[C]// Proceedings of the 33rd Chinese Control and Decision Conference(CCDC2021). Piscataway:IEEE, 2021:5954-5959.
[25]	LI Y, WANG Q, LIU R. Research on YOLOv3 Pedestrian Detection Algorithm Based on Channel Attention Mechanism[C]// Proceedings of the 1st IEEE International Conference on Computer Science,Electronic Information Engineering and Intelligent Control Technology(CEI2021). Piscataway:IEEE, 2021:229-232.
[26]	WANG X, HU H, ZHANG Y. Pedestrian Detection Based on Spatial Attention Module for Outdoor Video Surveillance[C]// Proceedings of IEEE 5th International Conference on Multimedia Big Data(BigMM2019). Piscataway:IEEE, 2019:247-251.
[27]	FU J, LIU J, TIAN H, et al. Dual Attention Network for Scene Segmentation[C]// Proceedings of the 32nd IEEE Conference on Computer Vision and Pattern Recognition(CVPR2019). Piscataway:IEEE 2019:3141-3149.
[28]	SHRIVASTAVA A, GUPTA A, GIRSHICK R. Training Region-Based Object Detectors with Online Hard Example Mining[C]// Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition(CVPR2016). Piscataway:IEEE, 2016:761-769.
[29]	ZHANG S, BENENSON R, SCHIELE B. Citypersons:A Diverse Dataset for Pedestrian Detection[C]// Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition(CVPR2017). Piscataway:IEEE, 2017:3213-3221.
[30]	DOLLAR P, WOJEK C, SCHIELE B,etal. Pedestrian Detection:An Evaluation of the State of the Art[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 34(4):743-761.
[31]	GAO F, CAI Y, DENG F, et al. Feature Alignment in Anchor-Free Object Detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8):3799-3810.
[32]	CHEN Y, YE J, WAN X. TF-YOLO:A Transformer-Fusion-Based YOLO Detector for Multimodal Pedestrian Detection in Autonomous Driving Scenes.[J]. World Electric Vehicle Journal. 2023; 14(12):352.
[33]	ZHANG S, WEN L, BIAN X, et al. Occlusion-Aware R-CNN:Detecting Pedestrians in a Crowd[C]// Proceedings of the 15th European Conference on Computer Vision(ECCV2018). Berlin:Springer, 2018:637-653.
[34]	TIAN Y, LUO P, WANG X, et al. DeepLearning Strong Parts for Pedestrian Detection[C]// Proceedings of the 15th IEEE International Conference on Computer Vision(ICCV). Piscataway:IEEE, 2015:1904-1912.
[35]	QIU T, WANG L, WANG P, et al. Research on Object Detection Algorithm Based on Improved YOLOv5[J]. Journal of Computer Engineering & Applications, 2022, 58(13):63-73.
[36]	CONG C, YANG Z, SONG Y, et al. Towards Enforcing Social Distancing Regulations with Occlusion-Aware Crowd Detection[C]// Proceedings of the 16th International Conference on Control,Automation,Robotics and Vision(ICARCV2020). Piscataway:IEEE, 2020:297-302.
[37]	SHAO S, ZHAO Z, LI B, et al. Crowdhuman:A Benchmark for Detecting Human in a Crowd(2023)[J/OL]. [2023-6-10].https://doi.org/10.48550/arXiv.1805.00123.
[38]	CHEN K, WANG J, PANG J, et al. MMDetection:OpenMmlab Detection Toolbox and Benchmark(2023)[J/OL]. [2023-6-17].https://doi.org/10.48550/arXiv.1906.07155.
[39]	KINGMA D P, BA J. Adam:A Method for Stochastic Optimization(2023)[J/OL]. [2023-6-22].https://doi.org/10.48550/arXiv.1412.6980.
[40]	LIN Z, PEI W, CHEN F, et al. Pedestrian Detection by Exemplar-Guided Contrastive Learning[J]. IEEE Transactions on Image Processing, 2022, 32:2003-2016
[41]	WANG X, XIAO T, JIANG Y, et al. Repulsion Loss:Detecting Pedestrians in a Crowd[C]// Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition(CVPR2018). Piscataway:IEEE, 2018:7774-7783.
[42]	ZHANG S, BENENSON R, OMRAN M, et al. How Far Are We From Solving Pedestrian Detection?[C]// Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition(CVPR2016). Piscataway:IEEE, 2016:1259-1267.
[43]	BRAZIL G, LIU X. Pedestrian Detection with Autoregressive Network Phases[C]// Proceedings of the 32nd IEEE Conference on Computer Vision and Pattern Recognition(CVPR2019). Piscataway:IEEE, 2019:7231-7240.
[44]	BRAZIL G, YIN X, LIU X. Illuminating Pedestrians via Simultaneous Detection & Segmentation[J]. IEEE International Conference on Computer Vision, 2017:4950-4959.
[45]	LIU W, ANGUELOV D, ERHAN D, et al. SSD:Single Shot Multibox Detector[C]// Proceedings of Computer Vision-ECCV 2016:14th European Conference. Berlin:Springer, 2016:21-37.
[46]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal Loss for Dense Object Detection(2023)[J/OL]. [2023-6-27].https://doi.org/10.48550/arXiv.1708.02002.
[47]	REN S, HE K, GIRSHICK R B, et al. Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39:1137-1149.

类别	特点概述	缺点	代表性方法
基于子部件模型的方法	基于子部件模型的方法通过分割和独立处理行人的各个部件,以及综合各部件的检测结果,适用于需要高精度和鲁棒性的应用场景。	复杂性高,需要大量计算资源,并且设计正确的部件分割可能具有挑战性,限制了模型在复杂场景下的适应能力。	CircleNet^[20], OR-CNN^[33]
基于注意力机制的方法	基于注意力机制的行人检测模型通过引入注意力机制来增强和校准行人特征,从而提升检测器在遮挡情况下的准确性和鲁棒性。	注意力分配可能不准确,增加了计算成本,而且对模型结构的依赖性较强,设计和优化相对复杂。	MGAN^[10], YOLOv3^[25], TF-YOLO^[32], YOLO-G^[35]

方法	R	Δ	HO	Δ	R+HO	Δ
Baseline	13.2		46.8		29.5
Baseline+SSA	12.5	0.7	43.6	3.2	26.2	3.3
Baseline+SSA+PrC(DAFC)	11.3	1.9	40.6	6.2	23.4	6.1
DAFC+NUHEM	10.7	2.5	39.6	7.2	22.6	6.9

方法	Scale	R/%	HO/%
ATT-self	×1	20.9	58.3
ATT-vbb	×1	16.4	57.3
ATT-part	×1	16.0	56.7
EGCL	×1	11.5	51.1
DAFC	×1	11.2	46.7

方法	R	HO
基线网络	11.9	45.6
OHEM	11.0	43.2
Focal loss	11.2	44.3
NUHEM	11.8	44.8

方法	Scale	R/%	HO/%
ATT-part	×1	16.0	56.7
CCFA-Net	×1	15.4	52.9
RepLoss	×1	13.2	56.9
ALFNet	×1	12.0	51.9
OR-CNN	×1	12.8	55.7
FC-Net	×1	13.5	44.3
EGCL	×1	10.9	46.4
DAFC+	×1	11.8	44.8
CCFA-Net	×1.3	13.4	50.0
RepLoss	×1.3	11.6	55.3
OR-CNN	×1.3	11.0	51.3
CircleNet	×1.3	11.8	50.2
FC-Net	×1.3	11.6	42.8
EGCL	×1.3	10.5	45.3
DAFC+	×1.3	10.7	39.6

基于特征校准的双注意力遮挡行人检测器

Dual attention pedestrian detector for occlusion scenario based on feature calibration

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 47

相关文章 15

Metrics

本文评价

推荐阅读 0

方法	R	HO	R+HO
DeepParts	11.9	60.0	22.8
ATT-vbb	10.3	45.2	18.2
Adaptive FasterRCNN	9.2	57.6	20.0
Bi-box	7.6	44.4	16.1
SDS-RCNN	7.4	58.6	19.7
AR-Ped	6.5	48.8	16.1
DAFC+	6.9	37.1	12.9

方法	mAP50	mAP50∶95
SSD	69.6	35.5
RetinaNet	76.1	45.7
Faster-RCNN	79.3	46.6
YOLOv5-l	80.2	50.4
YOLOX-l	81.0	49.0
YOLOv7	83.1	52.3
YOLOv8-l	86.3	56.5
DAFC+	89.8	57.2

[1]	曲家慧, 贺杰, 董文倩, 李云松, 张同振, 杨宇菲. 联合多尺度高低频信息融合的变化检测方法[J]. 西安电子科技大学学报, 2025, 52(1): 105-116.
[2]	陈可嘉, 张雨鹏, 林鸿熙. 句法感知与知识增强的方面级情感分析[J]. 西安电子科技大学学报, 2024, 51(5): 165-178.
[3]	衡红军, 喻龙威. 基于多尺度特征信息融合的时间序列异常检测[J]. 西安电子科技大学学报, 2024, 51(3): 203-214.
[4]	龚峻扬, 付卫红, 方厚章. SAR图像舰船目标检测的轻量化和特征增强研究[J]. 西安电子科技大学学报, 2024, 51(2): 96-106.
[5]	钟浩, 边山, 王春桃. 结合自注意力与卷积的真实场景图像篡改定位[J]. 西安电子科技大学学报, 2024, 51(1): 135-146.
[6]	张欣雨, 梁煜, 张为. 融合全局和局部信息的实时烟雾分割算法[J]. 西安电子科技大学学报, 2024, 51(1): 147-156.
[7]	黄河源, 慕彩红, 方云飞, 刘逸. 使用图负采样的图卷积神经网络推荐算法[J]. 西安电子科技大学学报, 2024, 51(1): 86-99.
[8]	杜明洋, 杜蒙, 潘继飞, 毕大平. 基于生成对抗网络的雷达脉内信号去噪与识别[J]. 西安电子科技大学学报, 2023, 50(6): 133-147.
[9]	张可涵,李红艳,刘文慧,王鹏. 面向流量预测的时间相关图卷积网络构建方法[J]. 西安电子科技大学学报, 2023, 50(5): 11-20.
[10]	范文同,李震宇,张涛,罗向阳. 基于隐写噪声深度提取的JPEG图像隐写分析[J]. 西安电子科技大学学报, 2023, 50(4): 157-169.
[11]	王侃, 王孟洋, 刘鑫, 田国强, 李川, 刘伟. 融合自注意力机制与CNN-BiGRU的事件检测[J]. 西安电子科技大学学报, 2022, 49(5): 181-188.
[12]	时云龙,袁文浩,胡少东,娄迎曦. 一种用于实时语音增强的卷积准循环网络[J]. 西安电子科技大学学报, 2022, 49(3): 183-190.
[13]	周鹏,杨军. 索引边缘几何卷积神经网络用于点云分类[J]. 西安电子科技大学学报, 2022, 49(2): 207-217.
[14]	李源,崔玉爽,王伟. 一种基于字词双通道网络的文本情感分析方法[J]. 西安电子科技大学学报, 2021, 48(6): 179-186.
[15]	于浩洋,尹良,李书芳,吕顺. 生成对抗网络小样本雷达调制信号识别算法[J]. 西安电子科技大学学报, 2021, 48(6): 96-104.