融合全局和局部信息的实时烟雾分割算法

doi:10.19665/j.issn1001-2400.20230405

摘要/Abstract

摘要：

针对烟雾形状不规则、呈半透明状且边界模糊导致烟雾分割困难的问题,提出一种融合全局和局部信息的双分支实时烟雾分割算法。该算法设计了轻量级的Transformer分支和卷积神经网络分支分别提取烟雾的全局特征和局部特征,Transformer分支和卷积神经网络分支共同作用,可以在充分学习烟雾的长距离像素依赖关系的同时保留烟雾细节信息,从而准确区分烟雾和背景像素,改善烟雾分割效果。同时该结构可以满足实际烟雾检测任务的实时性要求;基于多层感知机的解码器充分利用不同尺度的烟雾特征图,并进一步建模烟雾全局上下文信息,增强模型对多尺度烟雾的感知能力,从而提升烟雾分割精度;而且解码器结构简单,可以降低解码器部分的计算量。该算法在自建烟雾分割数据集上的平均交并比为92.88%,模型参数量为2.96 M,推理速度为56.94帧/s。该算法在公开数据集上的综合性能优于其他烟雾检测算法。实验结果表明,该算法分割烟雾的准确率高,推理速度快,可以满足实际烟雾检测任务的准确性和实时性需求。

关键词: 烟雾分割, Transformer, 卷积神经网络, 双分支

Abstract:

The smoke segmentation is challenging because the smoke is irregular and translucent and the boundary is fuzzy.A dual-branch real-time smoke segmentation algorithm based on global and local information is proposed to solve this problem.In this algorithm,a lightweight Transformer branch and a convolutional neural networks branch are designed to extract the global and local features of smoke respectively,which can fully learn the long-distance pixel dependence of smoke and retain the details of smoke.It can distinguish smoke and background accurately and improve the accuracy of smoke segmentation.It can satisfy the real-time requirement of the actual smoke detection tasks.The multilayer perceptron decoder makes full use of multi-scale smoke features and further models the global context information of smoke.It can enhance the perception of multi-scale smoke,and thus improve the accuracy of smoke segmentation.The simple structure can reduce the computation of the decoder.The algorithm reaches 92.88% mean intersection over union on the self-built smoke segmentation dataset with 2.96M parameters and a speed of 56.94 frames per second.The comprehensive performance of the proposed algorithm is better than that of other smoke detection algorithms on public dataset.Experimental results show that the algorithm has a high accuracy and fast inference speed.The algorithm can meet the accuracy and real-time requirements of actual smoke detection tasks.

Key words: smoke segmentation, Transformer, convolutional neural networks, dual-branch

中图分类号:

TP391.41

张欣雨, 梁煜, 张为. 融合全局和局部信息的实时烟雾分割算法[J]. 西安电子科技大学学报, 2024, 51(1): 147-156.

ZHANG Xinyu, LIANG Yu, ZHANG Wei. Real-time smoke segmentation algorithm combining global and local information[J]. Journal of Xidian University, 2024, 51(1): 147-156.

图/表 10

图1

图2

图3

图4

图5

图6

图7

表1

表2

表3

参考文献 28

[1]	秦瑞, 张为. 一种无锚框结构的多尺度火灾检测算法[J]. 西安电子科技大学学报, 2022, 49(6):111-119.
	QIN Rui, ZHANG Wei. Multi-Scale Fire Detection Algorithm with an Anchor Free Structure[J]. Journal of Xidian University, 2022, 49(6):111-119.
[2]	宁阳, 杜建超, 韩硕, 等. 改进DeeplabV3+的火焰分割与火情分析方法[J]. 西安电子科技大学学报, 2021, 48(5):38-46.
	NING Yang, DU Jianchao, HAN Shuo, et al. Fire Segmentation Based on the Improved DeeplabV3+ and the Analytical Method for Fire Development[J]. Journal of Xidian University, 2021, 48(5):38-46.
[3]	DENG X, YU Z, WANG L, et al. Smoke Image Segmentation Based on Color Model[J]. Journal on Innovation and Sustainability RISUS, 2015, 6(2):130-138. doi: 10.24212/2179-3565.2015v6i2p130-138
[4]	MA Z, CAO Y, SONG L, et al. A New Smoke Segmentation Method Based on Improved Adaptive Density Peak Clustering[J]. Applied Sciences, 2023, 13(3):1281. doi: 10.3390/app13031281
[5]	WANG Z, YANG P, LIANG H, et al. Semantic Segmentation and Analysis on Sensitive Parameters of Forest Fire Smoke Using Smoke-Unet and Landsat-8 Imagery[J]. Remote Sensing, 2022, 14(1):45. doi: 10.3390/rs14010045
[6]	RONNEBERGER O, FISCHER P, BROX T. U-Net:Convolutional Networks for Biomedical Image Segmentation[C]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Heidelberg: Springer, 2015:234-241.
[7]	YUAN F, SHI Y, ZHANG L, et al. A Cross-Scale Mixed Attention Network for Smoke Segmentation[J]. Digital Signal Processing, 2023, 134:103924. doi: 10.1016/j.dsp.2023.103924
[8]	ZHENG S, LU J, ZHAO H, et al. Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021:6881-6890.
[9]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16x16 Words:Transformers for Image Recognition at Scale[J]. International Conference on Learning Representations, 2021.
[10]	CHEN J, LU Y, YU Q, et al. TransUnet:Transformers Make Strong Encoders for Medical Image Segmentation(2021)[J/OL].[2021-02-08].https://arxiv.org/abs/2102.04306v1.
[11]	ZHENG Y, WANG Z, XU B, et al. Multi-Scale Semantic Segmentation for Fire Smoke Image Based on Global Information and U-Net[J]. Electronics, 2022, 11(17):2718. doi: 10.3390/electronics11172718
[12]	LIU Z, LIN Y, CAO Y, et al. Swin Transformer:Hierarchical Vision Transformer Using Shifted Windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021:10012-10022.
[13]	XIE E, WANG W, YU Z, et al. SegFormer:Simple and Efficient Design for Semantic Segmentation with Transformers[J]. Advances in Neural Information Processing Systems, 2021, 34:12077-12090.
[14]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need[C]// Advances in Neural Information Processing Systems 30(NIPS 2017). San Diego: NIPS, 2017:1-11.
[15]	ISLAM M A, JIA S, BRUCE N D B. How Much Position Information Do Convolutional Neural Networks Encoder(2020)[J/OL].[2020-01-22].https://arxiv.org/abs/2001.08248.
[16]	SANDLER M, HOWARD A, ZHU M, et al. Mobilenetv2:Inverted Residuals and Linear Bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018:4510-4520.
[17]	LIU Z, MAO H, WU C Y, et al. A Convnet for the 2020s[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022:11976-11986.
[18]	YU C, WANG J, PENG C, et al. Bisenet:Bilateral Segmentation Network for Real-Time Semantic Segmentation[C]//Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2018:325-341.
[19]	HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018:7132-7141.
[20]	GUO M H, LU C Z, HOU Q, et al. SegneXt:Rethinking Convolutional Attention Design for Semantic Segmentation[J]. Advances in Neural Information Processing Systems, 2022, 35:1140-1156.
[21]	GENG Z, GUO M H, CHEN H, et al. Is Attention Better Than Matrix Decomposition?(2021)[J/OL].[2021-09-09].https://arxiv.org/abs/2109.04553.
[22]	ZHAO H, SHI J, QI X, et al. Pyramid Scene Parsing Network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017:2881-2890.
[23]	CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation[C]//Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2018:801-818.
[24]	YUAN Y, CHEN X, CHEN X, et al. Object-Contextual Representations for Semantic Segmentation[C]// Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2020:173-190.
[25]	XIAO T, LIU Y, ZHOU B, et al. Unified Perceptual Parsing for Scene Understanding[C]//Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2018:418-434.
[26]	BESBES O, BENAZZA-BENYAHIA A. A Novel Video-Based Smoke Detection Method Based on Color Invariants[C]//2016 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP). Piscataway: IEEE, 2016:1911-1915.
[27]	赵敏, 张为, 王鑫, 等. 时空背景模型下结合多种纹理特征的烟雾检测[J]. 西安交通大学学报, 2018, 52(8):67-73.
	ZHAO Min, ZHANG Wei, WANG Xin, et al. A Smoke Detection Algorithm with Multi-Texture Feature Exploration Under a Spatio-Temporal Background Model[J]. Journal of Xi’an Jiaotong University, 2018, 52(8):67-73.
[28]	王浩远, 梁煜, 张为. 融合多分辨率表征的实时烟雾分割算法[J]. 浙江大学学报(工学版), 2021, 55(12):2334-2341.
	WHANG Haoyuan, LIANG Yu, ZHANG Wei. Real-Time Smoke Segmentation Algorithm Fused with Multi-Resolution Representation[J]. Journal of ZheJiang University(Engineering Science), 2021, 55(12):2334-2341.

算法	R_mIoU/%	Params/M	FPS/(帧·s^-1)
PSPNet	91.92	48.96	32.30
Deeplabv3+	92.65	43.58	26.13
OCRNet	92.56	70.35	15.83
SETR	89.61	318.31	9.91
Swin-T	92.75	60.00	19.31
文中算法	92.88	2.96	56.94

视频名称	文中算法		文献[28]		文献[27]		文献[26]
视频名称	R_TPR	R_TNR	R_TPR	R_TNR	R_TPR	R_TNR	R_TPR	R_TNR
sBehindtheFence	98.73	97.80	98.64	100.00	97.20	96.27	94.72	100.00
sBtFence	99.16	100.00	98.81	100.00	98.17	100.00	99.08	100.00
sMoky	98.44	100.00	98.27	100.00	99.68	100.00	86.23	100.00
sWasteBasket	99.52	97.15	99.50	96.84	97.18	98.36	99.89	92.60
sWindow	98.28	100.00	98.46	97.87	98.10	100.00	94.30	100.00
平均值	98.83	98.99	98.74	98.94	98.07	98.93	94.84	98.52

Transformer分支	CNN分支	特征融合模块	解码器	R_mIoU/%	Params/M	FPS/(帧·s^-1)
Swin-T	-	-	-	92.75	60.00	19.31
文中算法	-	-	-	91.70	9.87	49.42
文中算法没有使用该模块	√	-	-	82.82	0.99	95.34
文中算法	√	-	-	92.13	10.13	47.62
文中算法	√	√	-	92.45	10.20	46.44
文中算法	√	√	√	92.88	2.96	56.94