一种用于自动驾驶场景的轻量级语义分割网络

doi:10.19665/j.issn1001-2400.2023.01.014

摘要/Abstract

摘要：

在自动驾驶场景下,针对语义分割模型在车载硬件设备中部署时内存受限且算力不足的问题,需要设计一种较好权衡效率和精度的语义分割模型。采用单分支网络结构,设计了一个轻量级多尺度双向注意力网络。为了实现高效的特征提取,设计了一种轻量级卷积单元来构成网络的特征提取骨干。为了较好地定位和分割道路场景中尺度差异较大的物体,提出了一种多尺度双向注意力模块。它具有全局多尺度感受野,并且在沿一个方向编码通道注意力的同时保留了另一个方向的空间位置信息。基于该注意力模块,设计了跳跃注意力连接模块和特征注意力融合模块,使得输出特征兼具细节信息和语义信息。模型在Cityscapes数据集上以0.9M的参数量,取得了71.86%的平均交并比,同时在单个RTX2080Ti GPU下实现了88FPS的推理速度。实验结果表明,该模型能够实现较高的分割精度,适用于车载硬件下的部署和应用,具有一定的实用价值。

关键词: 自动驾驶, 语义分割, 轻量级网络, 注意力机制, 深度学习

Abstract:

In the autonomous driving scenario,aiming at the problem of limited memory and insufficient computing power when the semantic segmentation model is deployed in vehicle hardware devices,it is necessary to design a semantic segmentation model that can balance efficiency and accuracy.In this paper,a single-branch network structure is used to design a lightweight multi-scale bidirectional attention network.To achieve efficient feature extraction,a lightweight convolutional unit is designed to form the feature extraction backbone of the network.In order to better locate and segment objects with large scale differences in road scenes,a multi-scale bidirectional attention module is proposed which has a global multi-scale receptive field and encodes channel attention in one direction while preserving spatial location information in the other.Based on this attention module,a skip attention connection module and a feature attention fusion module are designed,so that the output features have both detailed information and semantic information.On the Cityscapes dataset,the model in this paper achieves an MIoU(Mean Intersection over Union) of 71.86% with a parameter size of 0.9M,and achieves an inference speed of 88FPS(Frames Per Second) under a single RTX2080Ti GPU.The test results on public datasets show that the model achieves a high segmentation accuracy and is suitable for deployment and application under in-vehicle hardware,which is of certain practical value.

Key words: automatic driving, semantic segmentation, lightweight network, attention mechanism, deep learning

中图分类号:

TP391.4

刘博翀, 蔡怀宇, 杨诗远, 李灏天, 汪毅, 陈晓冬. 一种用于自动驾驶场景的轻量级语义分割网络[J]. 西安电子科技大学学报, 2023, 50(1): 118-128.

LIU Bochong, CAI Huaiyu, YANG Shiyuan, LI Haotian, WANG Yi, CHEN Xiaodong. Lightweight semantic segmentation network for autonomous driving scenarios[J]. Journal of Xidian University, 2023, 50(1): 118-128.

图/表 11

图1

图2

图3

图4

表1

表2

表3

表4

图5

表5

图6

参考文献 31

[1]	汪梓艺, 苏育挺, 刘艳艳, 等. 一种改进DeeplabV3网络的烟雾分割算法[J]. 西安电子科技大学学报, 2019, 46(6):52-59.
	WANG Ziyi, SU Yuting, LIU Yanyan, et al. An Improved DeeplabV3 Network Smoke Segmentation Algorithm[J]. Journal of Xidian University, 2019, 46(6):52-59.
[2]	回海生, 张雪英, 吴泽林, 等. 一种主辅路径注意力补偿的脑卒中病灶分割方法[J]. 西安电子科技大学学报, 2021, 48(4):200-208.
	HUI Haisheng, ZHANG Xueying, WU Zelin, et al. Lightweight Image Super-Resolution with the Adaptive Weight Learning Network[J]. Journal of Xidian University, 2021, 48(4):200-208.
[3]	SHELHAMER E, LONG J, DARRELL T. Fully Convolutional Networks for Semantic Segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39:640-651. doi: 10.1109/TPAMI.2016.2572683 pmid: 27244717
[4]	LI Y, SHI T, ZHANG Y, et al. Learning Deep Semantic Segmentation Network under Multiple Weakly-Supervised Constraints for Cross-Domain Remote Sensing Image Semantic Segmentation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 175:20-33. doi: 10.1016/j.isprsjprs.2021.02.009
[5]	YUAN X, SHI J, GU L. A Review of Deep Learning Methods for Semantic Segmentation of Remote Sensing Imagery[J]. Expert Systems with Applications, 2021, 169:114417. doi: 10.1016/j.eswa.2020.114417
[6]	WANG Y, ZHOU Q, LIU J, et al. LEDNet:A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation[C]// Proceedings of the 2019 IEEE International Conference on Image Processing(ICIP).Piscataway:IEEE, 2019:1860-1864.
[7]	WANG Y, ZHOU Q, XIONG J, et al. ESNet:An Efficient Symmetric Network for Real-Time Semantic Segmentation[C]// Proceedings of the Chinese Conference on Pattern Recognition and Computer Vision(PRCV).Cham:Springer, 2019:41-52.
[8]	ROMERA E, ALVAREZ J M, BERGASA L M, et al. ERFNet:Efficient Residual Factorized Convnet for Real-Time Semantic Segmentation[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 19(1):263-272. doi: 10.1109/TITS.2017.2750080
[9]	WANG K, YANG J, YUAN S, et al. A Lightweight Network with Attention Decoder for Real-Time Semantic Segmentation[J]. The Visual Computer, 2021, 38(7):2329-2339. doi: 10.1007/s00371-021-02115-4
[10]	ZHUANG M, ZHONG X, GU D, et al. LRDNet:A Lightweight and Efficient Network with Refined Dual Attention Decoder for Real-Time Semantic Segmentation[J]. Neurocomputing, 2021, 459:349-360. doi: 10.1016/j.neucom.2021.07.019
[11]	YU C, WANG J, PENG C, et al. BiSeNet:Bilateral Segmentation Network for Real-Time Semantic Segmentation[C]// Proceedings of the European Conference on Computer Vision(ECCV).Cham:Springer, 2018:334-349.
[12]	WU T, TANG S, ZHANG R, et al. CGNet:A Light-weight Context Guided Network for Semantic Segmentation[J]. IEEE Transactions on Image Processing, 2021, 30:1169-1179. doi: 10.1109/TIP.83
[13]	GAO G, XU G, YU Y, et al. MSCFNet:A Lightweight Network with Multi-Scale Context Fusion for Real-Time Semantic Segmentation[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(12):25489-25499. doi: 10.1109/TITS.2021.3098355
[14]	YANG Q, CHEN T, FAN J, et al. EADNet:Efficient Asymmetric Dilated Network for Semantic Segmentation[C]// Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).Piscataway:IEEE, 2021:2315-2319.
[15]	HE K, ZHANG X, REN S, et al. Identity Mappings in Deep Residual Networks[C]// Proceedings of the European Conference on Computer Vision.Cham:Springer, 2016:630-645.
[16]	CHOLLET F. Xception:Deep Learning with Depthwise Separable Convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:1800-1807.
[17]	PASZKE A, CHAURASIA A, KIM S, et al. ENet:A Deep Neural Network Architecture for Real-Time Semantic Segmentation[J/OL]. [2016-06-07] https://doi.org/10.48550/arXiv.1606.02147.
[18]	ZHANG X, ZHOU X, LIN M, et al. ShuffleNet:An Extremely Efficient Convolutional Neural Network for Mobile Devices[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:6848-6856.
[19]	成磊, 王玥, 田春娜. 一种添加残差注意力机制的视觉目标跟踪算法[J]. 西安电子科技大学学报, 2020, 47(6):148-157.
	CHENG Lei, WANG Yue, TIAN Chunna. A Visual Object Tracking Algorithm with Residual Attention Mechanism[J]. Journal of Xidian University, 2020, 47(6):148-157.
[20]	宋建锋, 苗启广, 王崇晓, 等. 注意力机制的多尺度单目标跟踪算法[J]. 西安电子科技大学学报, 2021, 48(5):110-116.
	SONG Jianfeng, MIAO Qiguang, WANG Chongxiao, et al. Multi-Scale Single Target Tracking Algorithm Based on Attention Mechanism[J]. Journal of Xidian University, 2021, 48(5):110-116.
[21]	CAO Y, XU J, LIN S, et al. GCNet:Non-Local Networks Meet Squeeze-Excitation Networks and Beyond[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.Piscataway:IEEE, 2019:1971-1980.
[22]	常新旭, 张杨, 杨林, 等. 融合多头自注意力机制的语音增强方法[J]. 西安电子科技大学学报, 2020, 47(1):104-110
	CHANG Xinxu, ZHANG Yang, YANG Lin, et al. A Speech Enhancement Method Integrating Multi-Head Self-Attention Mechanism[J]. Journal of Xidian University, 2020, 47(1):104-110.
[23]	WOO S, PARK J, LEE J Y, et al. CBAM:Convolutional Block Attention Module[C]// Proceedings of the European Conference on Computer Vision(ECCV).Cham:Springer, 2018:3-19.
[24]	HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8):2011-2023. doi: 10.1109/TPAMI.2019.2913372 pmid: 31034408
[25]	TIAN Z, HE T, SHEN C, et al. Decoders Matter for Semantic Segmentation:Data-Dependent Decoding Enables Flexible Feature Aggregation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019:3121-3130.
[26]	CORDTS M, OMRAN M, RAMOS S, et al. The Cityscapes Dataset for Semantic Urban Scene Understanding[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:3213-3223.
[27]	KINGMA D P, BA J. Adam:A Method for Stochastic Optimization[J/OL].[2017-01-30]. https://doi.org/10.48550/arXiv.1412.6980.
[28]	REZATOFIGHI H, TSOI N, GWAK J, et al. Generalized Intersection over Union:A Metric and a Loss for Bounding Box Regression[C] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019:658-666.
[29]	LI H, XIONG P, FAN H, et al. DFANet:Deep Feature Aggregation for Real-Time Semantic Segmentation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019:9514-9523.
[30]	ZHAO H, QI X, SHEN X, et al. ICNet for Real-Time Semantic Segmentation on High-Resolution Images[C]// Proceedings of the European Conference on Computer Vision(ECCV).Cham:Springer, 2018:418-434.
[31]	HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:770-778.

阶段1	阶段2	阶段3	平均交并比/%	参数量/M	浮点运算数每秒/G
1	4	8	68.92	0.895	10.25
2	4	8	71.86	0.900	10.87
3	4	8	71.15	0.902	11.48
2	3	8	70.76	0.880	10.29
2	5	8	70.37	0.915	11.44
2	4	7	71.01	0.830	10.31
2	4	9	71.41	0.965	11.42

SACM	FAFM	平均交并比/%	参数量/M	浮点运算数每秒/G
—	—	69.94	0.832	10.05
—	√	70.80_+0.86	0.864	10.27
√	—	70.77_+0.83	0.871	10.77
√	√	71.86_+1.92	0.900	10.87

r	平均交并比/%	参数量/M	浮点运算数每秒/G
4	71.01	0.95	10.88
8	71.86	0.90	10.87
16	69.57	0.87	10.86

SACM	FAFM	平均交并比/%	参数量/M	浮点运算数每秒/G
—	—	70.37	0.848	10.85
—	SE	70.81_+0.42	0.851	10.85
SE	—	70.76_+0.37	0.851	10.85
SE	SE	71.20_+0.81	0.853	10.85
—	MBA	71.08_+0.69	0.874	10.86
MBA	—	71.13_+0.74	0.874	10.86
MBA	MBA	71.86_+1.47	0.900	10.87

算法	Backbone	平均交并比/%	参数量/M	浮点运算数每秒/G	帧数每秒
CGNet^[12]	—	65.99	0.50	6.98	83
LEDNet^[6]	—	69.02	0.92	11.46	71
ESNet^[7]	—	69.10	1.66	23.24	67
ERFNet^[8]	—	68.13	2.06	25.79	55
BiSeNet^[11]	Xception39^[15]	69.00	5.80	2.90	71
DFANet^[29]	Xception A	70.30	7.80	2.10	77
ICNet^[30]	Resnet50^[31]	69.50	26.13	9.63	37
LMBANet	—	71.86	0.90	10.87	88