融合全局和局部信息的实时烟雾分割算法

doi:10.19665/j.issn1001-2400.20230405

Abstract

Abstract:

The smoke segmentation is challenging because the smoke is irregular and translucent and the boundary is fuzzy.A dual-branch real-time smoke segmentation algorithm based on global and local information is proposed to solve this problem.In this algorithm,a lightweight Transformer branch and a convolutional neural networks branch are designed to extract the global and local features of smoke respectively,which can fully learn the long-distance pixel dependence of smoke and retain the details of smoke.It can distinguish smoke and background accurately and improve the accuracy of smoke segmentation.It can satisfy the real-time requirement of the actual smoke detection tasks.The multilayer perceptron decoder makes full use of multi-scale smoke features and further models the global context information of smoke.It can enhance the perception of multi-scale smoke,and thus improve the accuracy of smoke segmentation.The simple structure can reduce the computation of the decoder.The algorithm reaches 92.88% mean intersection over union on the self-built smoke segmentation dataset with 2.96M parameters and a speed of 56.94 frames per second.The comprehensive performance of the proposed algorithm is better than that of other smoke detection algorithms on public dataset.Experimental results show that the algorithm has a high accuracy and fast inference speed.The algorithm can meet the accuracy and real-time requirements of actual smoke detection tasks.

Key words: smoke segmentation, Transformer, convolutional neural networks, dual-branch

CLC Number:

TP391.41

ZHANG Xinyu, LIANG Yu, ZHANG Wei. Real-time smoke segmentation algorithm combining global and local information[J].Journal of Xidian University, 2024, 51(1): 147-156.

Figures/Tables 10

References 28

[1]	秦瑞, 张为. 一种无锚框结构的多尺度火灾检测算法[J]. 西安电子科技大学学报, 2022, 49(6):111-119.
	QIN Rui, ZHANG Wei. Multi-Scale Fire Detection Algorithm with an Anchor Free Structure[J]. Journal of Xidian University, 2022, 49(6):111-119.
[2]	宁阳, 杜建超, 韩硕, 等. 改进DeeplabV3+的火焰分割与火情分析方法[J]. 西安电子科技大学学报, 2021, 48(5):38-46.
	NING Yang, DU Jianchao, HAN Shuo, et al. Fire Segmentation Based on the Improved DeeplabV3+ and the Analytical Method for Fire Development[J]. Journal of Xidian University, 2021, 48(5):38-46.
[3]	DENG X, YU Z, WANG L, et al. Smoke Image Segmentation Based on Color Model[J]. Journal on Innovation and Sustainability RISUS, 2015, 6(2):130-138. doi: 10.24212/2179-3565.2015v6i2p130-138
[4]	MA Z, CAO Y, SONG L, et al. A New Smoke Segmentation Method Based on Improved Adaptive Density Peak Clustering[J]. Applied Sciences, 2023, 13(3):1281. doi: 10.3390/app13031281
[5]	WANG Z, YANG P, LIANG H, et al. Semantic Segmentation and Analysis on Sensitive Parameters of Forest Fire Smoke Using Smoke-Unet and Landsat-8 Imagery[J]. Remote Sensing, 2022, 14(1):45. doi: 10.3390/rs14010045
[6]	RONNEBERGER O, FISCHER P, BROX T. U-Net:Convolutional Networks for Biomedical Image Segmentation[C]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Heidelberg: Springer, 2015:234-241.
[7]	YUAN F, SHI Y, ZHANG L, et al. A Cross-Scale Mixed Attention Network for Smoke Segmentation[J]. Digital Signal Processing, 2023, 134:103924. doi: 10.1016/j.dsp.2023.103924
[8]	ZHENG S, LU J, ZHAO H, et al. Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021:6881-6890.
[9]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16x16 Words:Transformers for Image Recognition at Scale[J]. International Conference on Learning Representations, 2021.
[10]	CHEN J, LU Y, YU Q, et al. TransUnet:Transformers Make Strong Encoders for Medical Image Segmentation(2021)[J/OL].[2021-02-08].https://arxiv.org/abs/2102.04306v1.
[11]	ZHENG Y, WANG Z, XU B, et al. Multi-Scale Semantic Segmentation for Fire Smoke Image Based on Global Information and U-Net[J]. Electronics, 2022, 11(17):2718. doi: 10.3390/electronics11172718
[12]	LIU Z, LIN Y, CAO Y, et al. Swin Transformer:Hierarchical Vision Transformer Using Shifted Windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021:10012-10022.
[13]	XIE E, WANG W, YU Z, et al. SegFormer:Simple and Efficient Design for Semantic Segmentation with Transformers[J]. Advances in Neural Information Processing Systems, 2021, 34:12077-12090.
[14]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need[C]// Advances in Neural Information Processing Systems 30(NIPS 2017). San Diego: NIPS, 2017:1-11.
[15]	ISLAM M A, JIA S, BRUCE N D B. How Much Position Information Do Convolutional Neural Networks Encoder(2020)[J/OL].[2020-01-22].https://arxiv.org/abs/2001.08248.
[16]	SANDLER M, HOWARD A, ZHU M, et al. Mobilenetv2:Inverted Residuals and Linear Bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018:4510-4520.
[17]	LIU Z, MAO H, WU C Y, et al. A Convnet for the 2020s[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022:11976-11986.
[18]	YU C, WANG J, PENG C, et al. Bisenet:Bilateral Segmentation Network for Real-Time Semantic Segmentation[C]//Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2018:325-341.
[19]	HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018:7132-7141.
[20]	GUO M H, LU C Z, HOU Q, et al. SegneXt:Rethinking Convolutional Attention Design for Semantic Segmentation[J]. Advances in Neural Information Processing Systems, 2022, 35:1140-1156.
[21]	GENG Z, GUO M H, CHEN H, et al. Is Attention Better Than Matrix Decomposition?(2021)[J/OL].[2021-09-09].https://arxiv.org/abs/2109.04553.
[22]	ZHAO H, SHI J, QI X, et al. Pyramid Scene Parsing Network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017:2881-2890.
[23]	CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation[C]//Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2018:801-818.
[24]	YUAN Y, CHEN X, CHEN X, et al. Object-Contextual Representations for Semantic Segmentation[C]// Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2020:173-190.
[25]	XIAO T, LIU Y, ZHOU B, et al. Unified Perceptual Parsing for Scene Understanding[C]//Proceedings of the European Conference on Computer Vision(ECCV). Heidelberg: Springer, 2018:418-434.
[26]	BESBES O, BENAZZA-BENYAHIA A. A Novel Video-Based Smoke Detection Method Based on Color Invariants[C]//2016 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP). Piscataway: IEEE, 2016:1911-1915.
[27]	赵敏, 张为, 王鑫, 等. 时空背景模型下结合多种纹理特征的烟雾检测[J]. 西安交通大学学报, 2018, 52(8):67-73.
	ZHAO Min, ZHANG Wei, WANG Xin, et al. A Smoke Detection Algorithm with Multi-Texture Feature Exploration Under a Spatio-Temporal Background Model[J]. Journal of Xi’an Jiaotong University, 2018, 52(8):67-73.
[28]	王浩远, 梁煜, 张为. 融合多分辨率表征的实时烟雾分割算法[J]. 浙江大学学报(工学版), 2021, 55(12):2334-2341.
	WHANG Haoyuan, LIANG Yu, ZHANG Wei. Real-Time Smoke Segmentation Algorithm Fused with Multi-Resolution Representation[J]. Journal of ZheJiang University(Engineering Science), 2021, 55(12):2334-2341.

算法	R_mIoU/%	Params/M	FPS/(帧·s^-1)
PSPNet	91.92	48.96	32.30
Deeplabv3+	92.65	43.58	26.13
OCRNet	92.56	70.35	15.83
SETR	89.61	318.31	9.91
Swin-T	92.75	60.00	19.31
文中算法	92.88	2.96	56.94

视频名称	文中算法		文献[28]		文献[27]		文献[26]
视频名称	R_TPR	R_TNR	R_TPR	R_TNR	R_TPR	R_TNR	R_TPR	R_TNR
sBehindtheFence	98.73	97.80	98.64	100.00	97.20	96.27	94.72	100.00
sBtFence	99.16	100.00	98.81	100.00	98.17	100.00	99.08	100.00
sMoky	98.44	100.00	98.27	100.00	99.68	100.00	86.23	100.00
sWasteBasket	99.52	97.15	99.50	96.84	97.18	98.36	99.89	92.60
sWindow	98.28	100.00	98.46	97.87	98.10	100.00	94.30	100.00
平均值	98.83	98.99	98.74	98.94	98.07	98.93	94.84	98.52

Transformer分支	CNN分支	特征融合模块	解码器	R_mIoU/%	Params/M	FPS/(帧·s^-1)
Swin-T	-	-	-	92.75	60.00	19.31
文中算法	-	-	-	91.70	9.87	49.42
文中算法没有使用该模块	√	-	-	82.82	0.99	95.34
文中算法	√	-	-	92.13	10.13	47.62
文中算法	√	√	-	92.45	10.20	46.44
文中算法	√	√	√	92.88	2.96	56.94

Real-time smoke segmentation algorithm combining global and local information

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 28

Related Articles 15

Metrics

Comments

Recommended 0

[1]	HUANG Heyuan, MU Caihong, FANG Yunfei, LIU Yi. Graph convolution neural network for recommendation using graph negative sampling [J]. Journal of Xidian University, 2024, 51(1): 86-99.
[2]	ZHONG Hao, BIAN Shan, WANG Chuntao. Real world image tampering localization combining the self-attention mechanism and convolutional neural networks [J]. Journal of Xidian University, 2024, 51(1): 135-146.
[3]	DU Mingyang, DU Meng, PAN Jifei, BI Daping. Generative adversarial model for radar intra-pulse signal denoising and recognition [J]. Journal of Xidian University, 2023, 50(6): 133-147.
[4]	WANG Kan, WANG Mengyang, LIU Xin, TIAN Guoqiang, LI Chuan, LIU Wei. Event detection by combining self-attention and CNN-BiGRU [J]. Journal of Xidian University, 2022, 49(5): 181-188.
[5]	ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network [J]. Journal of Xidian University, 2021, 48(5): 15-22.
[6]	HUI Haisheng,ZHANG Xueying,WU Zelin,LI Fenglian. Method for stroke lesion segmentation using the primary-auxiliary path attention compensation network [J]. Journal of Xidian University, 2021, 48(4): 200-208.
[7]	GUO Zekun,TIAN Long,HAN Ning,WANG Penghui,LIU Hongwei,CHEN Bo. Radar HRRP based few-shot target recognition with CNN-SSD [J]. Journal of Xidian University, 2021, 48(2): 7-14.
[8]	LI Kunlun,ZHANG Lu,XU Hongke,SONG Huansheng. Waveletdomain dilated network for fast low-dose CT image reconstruction [J]. Journal of Xidian University, 2020, 47(4): 86-93.
[9]	DANG Jisheng,YANG Jun. 3D model recognition and segmentation based on multi-feature fusion [J]. Journal of Xidian University, 2020, 47(4): 149-157.
[10]	WANG Jijun,HAO Ziyu,LI Hongliang. Optimization of memory access for the convolutional neural network training [J]. Journal of Xidian University, 2020, 47(2): 98-107.
[11]	ZHU Suya,DU Jianchao,LI Yunsong,WANG Xiaopeng. Method for bridge crack detection based on the U-Net convolutional networks [J]. Journal of Xidian University, 2019, 46(4): 35-42.
[12]	JIANG Xingguo,WAN Jinzhao,CAI Xiaodong,LI Haiou,CAO Yi. Algorithm for identification of fine-grained vehicles based on singular value decomposition and central metric [J]. Journal of Xidian University, 2019, 46(3): 82-88.
[13]	LIANG Xiaoxi, CAI Xiaodong, WANG Meng, KU Haohua. Face recognition method for integrating the nested residual CNN and angular metric [J]. Journal of Xidian University, 2018, 45(6): 144-149.
[14]	ZHAO Boran;ZHANG Li;SHI Guangming;HUANG Rong;XU Xinran. Design of the programmable neural network processor based on the transport triggered architecture [J]. Journal of Xidian University, 2018, 45(4): 92-98.
[15]	XIN Peng;XU Yuelei;MA Shiping;LI Shuai;LV Chao. Airport detection combining region proposal networks and adaptive pooling networks [J]. Journal of Xidian University, 2018, 45(3): 102-108.