基于雷达和视频融合的目标检测

doi:10.16180/j.cnki.issn1007-7820.2024.08.001

摘要/Abstract

摘要：

基于视频的目标检测在恶劣天气情况下识别效果较差,故需弥补视频缺陷、提高检测框架的鲁棒性。针对此问题,文中设计了一个基于雷达和视频融合的目标检测框架,利用YOLOv5(You Only Look Once version 5)网络获得图片特征图与图片检测框,利用基于密度的聚类获得雷达检测框,并将雷达数据进行编码,得到基于雷达信息的目标检测结果。最后将两者的检测框叠加得到新ROI(Region of Interest),并得到融合雷达信息后的分类向量,提高了在极端天气下检测的准确率。实验结果表明,该框架的mAP(mean Average Precision)达到了60.07%,且参数量仅为7.64×10⁶,表明该框架具有轻量级、计算速度快、鲁棒性高等特点,可以被广泛应用于嵌入式与移动端平台。

关键词: 传感器融合, 雷达信号处理, 雷达特征图提取, DBSCAN, 卡尔曼滤波, 目标检测, YOLOv5, R-CNN

Abstract:

The object detection based on video has the problem of poor recognition effect in bad weather, so it is necessary to make up for video defects and improve the robustness of detection framework. In view of this problem, this study designs an object detection framework based on radar and video fusion. YOLOv5 (You Only Look Once version 5) network is used to obtain image feature map and image detection frame, density-based clustering is used to obtain radar detection frame, and radar data is encoded to get object detection results based on radar information. Finally, the detection boxes of the two are superimposed to obtain a new ROI (Region of Interest), and the classification vector after fusion radar information is obtained, which improves the detection accuracy in extreme weather. The experimental results show that the mAP(mean Average Precision) of the framework reaches 60.07%, and the parameter number is only 7.64×10⁶, which indicates that the framework has the characteristics of lightweight, fast computing speed and high robustness, and can be widely used in embedded and mobile platforms.

Key words: sensor fusion, radar signal processing, radar feature map extraction, DBSCAN, Kalman filter, object detection, YOLOv5, R-CNN

中图分类号:

TP391

朱勇, 黄永明, 何幸. 基于雷达和视频融合的目标检测[J]. 电子科技, 2024, 37(8): 1-7.

ZHU Yong, HUANG Yongming, HE Xing. Research on Object Detection Based on Radar and Video Fusion[J]. Electronic Science and Technology, 2024, 37(8): 1-7.

图/表 15

图1

图2

图3

图4

表1

图5

图6

表2

图7

表3

表4

表5

表6

图8

图9

参考文献 21

[1]	Nobis F, Geisslinger M, Weber M, et al. A deep learning-based radar and camera sensor fusion architecture for object detection[EB/OL].(2020-05-15) [2022-11-02] https://browse.arxiv.org/pdf/2005.07431v1.pdf.
[2]	Liu P, Yu G, Wang Z, et al. Object classification based onenhanced evidence theory:Radar-vision fusion approach for roadside application[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71(1):1-12.
[3]	Li K, Yang X, Luo Y, et al. Road geometry perception without accurate positioning and lane information[J]. IET Intelligent Transport Systems, 2022, 16(7):940-957.
[4]	Zhu Y, Wang T, Zhu S. Adaptive multi-pedestrian tracking by multisensor:Track-to-track fusion using monocular 3D detection and MMW radar[J]. Remote Sensing, 2022, 14(8):1837-1838.
[5]	Lin C, Tian D, Duan X, et al. CL3D:Cameralidar 3D object detection with point feature enhancement and point-guided fusion[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10):18040-18050.
[6]	郑雪梅. 基于雷视融合的高速公路多元感知监控技术研究[J]. 现代信息科技, 2022, 6(24):159-162.
	Zheng Xuemei. Research on multi-sensing monitoring technology of highway based on fusion of radar and vision[J]. Modern Information Technology, 2022, 6(24):159-162.
[7]	Krishna K, Murty M N. Genetic K-means algorithm[J]. IEEE Transactions on Systems, 1999, 29(3):433-439.
[8]	Ahmed K N, Razak T A. An overview of various improve-ments of DBSCAN algorithm in clustering spatial databases[J]. International Journal of Advanced Research in Computer and Communication Engineering, 2016, 5(2):360-363.
[9]	Chui C K, Chen G. Kalman filtering>[M]. Berlin: Springer International Publishing, 2017:86-92.
[10]	Zhang Z. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11):1330-1334.
[11]	Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]. Columbus: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014:580-587.
[12]	Bappy J H, Roy-Chowdhury A K. CNN based region proposals for efficient object detection[C]. Phoenix: IEEE International Conference on Image Processing, 2016:3658-3662.
[13]	Uijlings J R R, Van De Sande K E A, Gevers T, et al. Selective search for object recognition[J]. International Journal of Computer Vision, 2013, 104(1):154-171.
[14]	He K, Gkioxari G, Dollár P, et al. Mask R-CNN[C]. Venice: Proceedings of the IEEE International Conference on Computer Vision, 2017:2980-2988.
[15]	Noble W S. What is a support vector machine?[J]. Nature Biotechnology, 2006, 24(12):1565-1567. doi: 10.1038/nbt1206-1565 pmid: 17160063
[16]	程长文, 陈玮, 陈劲宏, 等. 改进YOLO的口罩佩戴实时检测方法[J]. 电子科技, 2023, 36(2):73-80.
	Cheng Changwen, Chen Wei, Chen Jinhong, et al. YOLO- improve detection method of real-time mask wearing[J]. Electronic Science and Technology, 2023, 36(2):73-80.
[17]	Caesar H, Bankiti V, Lang A H, et al. Nuscenes:A multi-modal dataset for autonomous driving[C]. Seattle: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020:11621-11631.
[18]	Gruber T, Julca-Aguilar F, Bijelic M, et al. Gated2depth:Real-time dense lidar from gated images[EB/OL].(2019-02-13) [2022-11-04] https://arxiv.org/abs/1902.04997.
[19]	Nobis F, Shafiei E, Karle P, et al. Radar voxel fusion for 3D object detection[J]. Applied Sciences, 2021, 11(12):5598-5599.
[20]	Yadav R, Vierling A, Berns K, et al. Radar RGB fusion for robust object detection in autonomous vehicle[C]. Abu Dhabi: IEEE International Conference on Image Processing, 2020:1986-1990.
[21]	Dong X, Zhuang B, Mao Y, et al. Radar camera fusion via representation learning in autonomous driving[C]. Nashville: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021:1672-1681.

组成部分	Params/10⁶	FLOPs/B
YOLOv5-s	7.10	12.60
雷达算法	0.33	0.29
R-CNN子网	0.21	0.19
框架整体	7.64	13.08

设备名称	型号与参数
GPU	NVIDIA Tesla P100-16 GB
GPU带宽	732.16 GB·s^-1
显卡内存	16 GB
每秒浮点运算次数	5.18 TFLOPS
CPU	Intel(R) Xeon(R) Platinum 8163
可使用CPU数	12
可使用内存	64 GB
硬盘	INSPUR
硬盘带宽	281.31 MB·s^-1

置信度阈值	YOLOv5	融合框架
0.1	82.9%	87.8%
0.2	80.6%	84.1%
0.3	79.2%	82.3%
0.4	76.1%	80.8%
0.5	74.9%	78.1%

置信度阈值	YOLOv5	融合框架
0.1	70.9%	78.7%
0.2	63.0%	74.1%
0.3	50.2%	69.3%
0.4	40.1%	65.8%
0.5	31.9%	58.1%

雷达与视频融合框架	mAP
CRF-Net(AF)^[1]	44.85%
RVF-Net^[19]	54.86%
GRF-Net(AF,GRF)^[1]	55.99%
本文框架	60.07%
BIRANet^[20]	67.40%
AssociationNet^[21]	68.10%