基于雷达和视频融合的目标检测

doi:10.16180/j.cnki.issn1007-7820.2024.08.001

Abstract

Abstract:

The object detection based on video has the problem of poor recognition effect in bad weather, so it is necessary to make up for video defects and improve the robustness of detection framework. In view of this problem, this study designs an object detection framework based on radar and video fusion. YOLOv5 (You Only Look Once version 5) network is used to obtain image feature map and image detection frame, density-based clustering is used to obtain radar detection frame, and radar data is encoded to get object detection results based on radar information. Finally, the detection boxes of the two are superimposed to obtain a new ROI (Region of Interest), and the classification vector after fusion radar information is obtained, which improves the detection accuracy in extreme weather. The experimental results show that the mAP(mean Average Precision) of the framework reaches 60.07%, and the parameter number is only 7.64×10⁶, which indicates that the framework has the characteristics of lightweight, fast computing speed and high robustness, and can be widely used in embedded and mobile platforms.

Key words: sensor fusion, radar signal processing, radar feature map extraction, DBSCAN, Kalman filter, object detection, YOLOv5, R-CNN

CLC Number:

TP391

ZHU Yong, HUANG Yongming, HE Xing. Research on Object Detection Based on Radar and Video Fusion[J].Electronic Science and Technology, 2024, 37(8): 1-7.

Figures/Tables 15

Figure 1.

Figure 2.

Figure 3.

Figure 4.

Table 1.

Figure 5.

Figure 6.

Table 2.

Figure 7.

Table 3.

Table 4.

Table 5.

Table 6.

Figure 8.

Figure 9.

References 21

[1]	Nobis F, Geisslinger M, Weber M, et al. A deep learning-based radar and camera sensor fusion architecture for object detection[EB/OL].(2020-05-15) [2022-11-02] https://browse.arxiv.org/pdf/2005.07431v1.pdf.
[2]	Liu P, Yu G, Wang Z, et al. Object classification based onenhanced evidence theory:Radar-vision fusion approach for roadside application[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71(1):1-12.
[3]	Li K, Yang X, Luo Y, et al. Road geometry perception without accurate positioning and lane information[J]. IET Intelligent Transport Systems, 2022, 16(7):940-957.
[4]	Zhu Y, Wang T, Zhu S. Adaptive multi-pedestrian tracking by multisensor:Track-to-track fusion using monocular 3D detection and MMW radar[J]. Remote Sensing, 2022, 14(8):1837-1838.
[5]	Lin C, Tian D, Duan X, et al. CL3D:Cameralidar 3D object detection with point feature enhancement and point-guided fusion[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10):18040-18050.
[6]	郑雪梅. 基于雷视融合的高速公路多元感知监控技术研究[J]. 现代信息科技, 2022, 6(24):159-162.
	Zheng Xuemei. Research on multi-sensing monitoring technology of highway based on fusion of radar and vision[J]. Modern Information Technology, 2022, 6(24):159-162.
[7]	Krishna K, Murty M N. Genetic K-means algorithm[J]. IEEE Transactions on Systems, 1999, 29(3):433-439.
[8]	Ahmed K N, Razak T A. An overview of various improve-ments of DBSCAN algorithm in clustering spatial databases[J]. International Journal of Advanced Research in Computer and Communication Engineering, 2016, 5(2):360-363.
[9]	Chui C K, Chen G. Kalman filtering>[M]. Berlin: Springer International Publishing, 2017:86-92.
[10]	Zhang Z. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11):1330-1334.
[11]	Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]. Columbus: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014:580-587.
[12]	Bappy J H, Roy-Chowdhury A K. CNN based region proposals for efficient object detection[C]. Phoenix: IEEE International Conference on Image Processing, 2016:3658-3662.
[13]	Uijlings J R R, Van De Sande K E A, Gevers T, et al. Selective search for object recognition[J]. International Journal of Computer Vision, 2013, 104(1):154-171.
[14]	He K, Gkioxari G, Dollár P, et al. Mask R-CNN[C]. Venice: Proceedings of the IEEE International Conference on Computer Vision, 2017:2980-2988.
[15]	Noble W S. What is a support vector machine?[J]. Nature Biotechnology, 2006, 24(12):1565-1567. doi: 10.1038/nbt1206-1565 pmid: 17160063
[16]	程长文, 陈玮, 陈劲宏, 等. 改进YOLO的口罩佩戴实时检测方法[J]. 电子科技, 2023, 36(2):73-80.
	Cheng Changwen, Chen Wei, Chen Jinhong, et al. YOLO- improve detection method of real-time mask wearing[J]. Electronic Science and Technology, 2023, 36(2):73-80.
[17]	Caesar H, Bankiti V, Lang A H, et al. Nuscenes:A multi-modal dataset for autonomous driving[C]. Seattle: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020:11621-11631.
[18]	Gruber T, Julca-Aguilar F, Bijelic M, et al. Gated2depth:Real-time dense lidar from gated images[EB/OL].(2019-02-13) [2022-11-04] https://arxiv.org/abs/1902.04997.
[19]	Nobis F, Shafiei E, Karle P, et al. Radar voxel fusion for 3D object detection[J]. Applied Sciences, 2021, 11(12):5598-5599.
[20]	Yadav R, Vierling A, Berns K, et al. Radar RGB fusion for robust object detection in autonomous vehicle[C]. Abu Dhabi: IEEE International Conference on Image Processing, 2020:1986-1990.
[21]	Dong X, Zhuang B, Mao Y, et al. Radar camera fusion via representation learning in autonomous driving[C]. Nashville: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021:1672-1681.

组成部分	Params/10⁶	FLOPs/B
YOLOv5-s	7.10	12.60
雷达算法	0.33	0.29
R-CNN子网	0.21	0.19
框架整体	7.64	13.08

设备名称	型号与参数
GPU	NVIDIA Tesla P100-16 GB
GPU带宽	732.16 GB·s^-1
显卡内存	16 GB
每秒浮点运算次数	5.18 TFLOPS
CPU	Intel(R) Xeon(R) Platinum 8163
可使用CPU数	12
可使用内存	64 GB
硬盘	INSPUR
硬盘带宽	281.31 MB·s^-1

置信度阈值	YOLOv5	融合框架
0.1	82.9%	87.8%
0.2	80.6%	84.1%
0.3	79.2%	82.3%
0.4	76.1%	80.8%
0.5	74.9%	78.1%

置信度阈值	YOLOv5	融合框架
0.1	70.9%	78.7%
0.2	63.0%	74.1%
0.3	50.2%	69.3%
0.4	40.1%	65.8%
0.5	31.9%	58.1%

雷达与视频融合框架	mAP
CRF-Net(AF)^[1]	44.85%
RVF-Net^[19]	54.86%
GRF-Net(AF,GRF)^[1]	55.99%
本文框架	60.07%
BIRANet^[20]	67.40%
AssociationNet^[21]	68.10%

Research on Object Detection Based on Radar and Video Fusion

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 21

Related Articles 15

Metrics

Comments

Recommended 0

雷达与视频融合框架	Params/10⁶	FLOPs/B
CRF-Net^[1]	270.00	481.10
BIRANet^[20]	164.00	291.20
AssociationNet^[21]	214.00	377.90
本文框架	7.64	13.08

[1]	XIE Xijun, LI Feifei. Non-Local Support Attention Network for Few-Shot Object Detection [J]. Electronic Science and Technology, 2024, 37(8): 75-83.
[2]	LI Jun, LIU Hu, YANG Haima, WANG Yuan, XU Wencheng, HUANG Hongxin. Research on AGV Path Fusion of Object Detection and DWA Algorithm [J]. Electronic Science and Technology, 2024, 37(7): 33-42.
[3]	HE Xing, HUANG Yongming, ZHU Yong. Pavement Pothole Detection Method Based on Improved YOLOv5 [J]. Electronic Science and Technology, 2024, 37(7): 53-59.
[4]	YE Yuxin, JU Zhiyong, LAI Ying. Traffic Sign Detection Algorithm Incorporating Receptive Field Enhancement Module and Attention Mechanism [J]. Electronic Science and Technology, 2024, 37(6): 8-16.
[5]	XIA Rongcheng, LIU Deer. Vehicle Detection and Analysis in Urban Waterlogging Area Based on Deep Learning [J]. Electronic Science and Technology, 2024, 37(5): 18-24.
[6]	MA Wenjie, ZHANG Xuanxiong. Research on Blind Roads and Obstacle Recognition Algorithm Based on Deep Learning [J]. Electronic Science and Technology, 2024, 37(3): 75-83.
[7]	WANG Xiaozhu,YU Lianzhi. Small Object Detection Based on Convolution and Self-Attention of Aggregation [J]. Electronic Science and Technology, 2024, 37(2): 14-22.
[8]	BAI Rui,REN Zhu. Estimation and Performance Analysis of Unscented Kalman Filter with Randomly Missing Measurements [J]. Electronic Science and Technology, 2024, 37(2): 23-29.
[9]	HU Yongyang,LI Miao,MENG Fankai,ZHANG Feng,MENG Yiwei,SONG Yukun. Structured Compression and Acceleration of Network Based on Tiny-YOLOv3 [J]. Electronic Science and Technology, 2023, 36(8): 43-48.
[10]	WANG Changsong,CHEN Hui,WANG Licheng,QU Feng. Abnormal Data Detection and SOC Estimation Algorithm for Lithium Battery Considering Measured Outliers [J]. Electronic Science and Technology, 2023, 36(5): 34-40.
[11]	LIN Zhipeng,SUN Xiaohui,WEN Chenglin. An Extended Dimension Kalman Filter Method Based on Additive Hidden Variables [J]. Electronic Science and Technology, 2023, 36(5): 47-54.
[12]	SHI Jianke,QIAO Meiying,LI Bingfeng,ZHAO Yan. Underwater Occlusion Target Detection Algorithm Based on Attention Mechanism [J]. Electronic Science and Technology, 2023, 36(5): 62-70.
[13]	CUI Zhuodong,CHEN Wei,YIN Zhong. Helmet Wearing Detection Based on Enhanced Feature Fusion Network [J]. Electronic Science and Technology, 2023, 36(4): 44-51.
[14]	WANG Xinyue,YU Huimin,HU Luning. Cooperative Localization of IMMKF and Chan-Taylor Algorithm [J]. Electronic Science and Technology, 2023, 36(12): 1-8.
[15]	CHEN Ziang,LIU Na,YUAN Ye,LI Qingdu,WAN Lihong. Chinese License Plate Detection and Recognition in Unconstrained Scenarios Based on YOLO [J]. Electronic Science and Technology, 2023, 36(10): 1-8.