一种注意力机制的多波段图像特征级融合方法

doi:10.19665/j.issn1001-2400.2020.01.017

摘要/Abstract

摘要：

针对多波段同步融合图像普遍存在的清晰度不高、图像细节不丰富的问题,提出一种基于注意力机制生成对抗网络的图像特征级融合方法。首先,利用多波段特征图与其均值的差值构建注意力权重图,通过特征图与注意力权重图的点乘和相加获得特征增强图,以此构建特征增强模块;其次,设计特征级融合模块,将多波段特征增强图连接,通过归一化、上采样、卷积等操作重构融合图像;最后,将特征增强模块和特征融合模块级联建立生成器,并以VGG-16作为判别器构建生成对抗网络,以实现多波段图像端到端融合。实验结果表明,与当前经典的融合方法相比,所提出方法的平均梯度最为突出,验证了该方法的有效性。

关键词: 图像融合, 深度学习, 多波段图像, 特征级融合, 注意力机制, 生成对抗网络

Abstract:

Aiming at the low definition and poor details of synchronous multi-band image fusion, a novel method based on attention generative adversarial networks is proposed. First, the attention weight map is constructed using the difference between the multi-band feature map and its mean, then the feature enhancement map is obtained by the point multiplication and addition of the feature map and the attention weight map to construct the feature enhancement module. Second, the feature-level fusion module is designed, which connects the multi-band feature enhancement map and reconstructs the fused image through normalization, upsampling, convolution, etc. Finally, the feature enhancement module and the feature-level fusion module are cascaded to build the generator, and the VGG-16 is used as a discriminator to establish a Generative Adversarial Network, thereby implementing multi-band image end-to-end fusion. Experimental results show that the proposed fusion method can lead to the most prominent average gradient compared with classical fusion methods, and that the effectiveness of the proposed method is verified.

Key words: image fusion, deep learning, multi-band image, feature-level fusion, attention, generative adversarial networks

中图分类号:

TP391

杨晓莉,蔺素珍. 一种注意力机制的多波段图像特征级融合方法[J]. 西安电子科技大学学报, 2020, 47(1): 120-127.

YANG Xiaoli,LIN Suzhen. Method for multi-band image feature-level fusion based on the attention mechanism[J]. Journal of Xidian University, 2020, 47(1): 120-127.

图/表 10

图1

图2

图3

图4

图5

图6

图7

表1

图8

图9

参考文献 15

[1]	MA J Y, MA Y, LI C . Infrared and Visible Image Fusion Methods and Applications: a Survey[J]. Information Fusion, 2019,45:153-178.
[2]	LIU Y, CHEN X, CHENG J , et al. Infrared and Visible Image Fusion with Convolutional Neural Networks[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2018,16(3):1850018.
[3]	LI H, WU X J . DenseFuse: a Fusion Approach to Infrared and Visible Image[J]. IEEE Transaction on Image Processing, 2019,28(5):2614-2623.
[4]	JADERBERG M, SIMONYAN K, ZISSERMAN A , et al. Spatial Transformer Networks[C]//Advances in Neural Information Processing Systems 28: 2015. Vancouver: Neural Information Processing Systems Foundation, 2015: 2017-2025.
[5]	HU J, SHEN L, SUN G . Squeeze-and-excitation Networks[C]//Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2018: 7132-7141.
[6]	WANG F, JIANG M, QIAN C , et al. Residual Attention Network for Image Classification[C]//Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6450-6458.
[7]	DUAN C, WANG S, HUANG Q , et al. Feature Level MRI Fusion Based on 3D Dual Tree Compactly Supported Shearlet Transform[J]. Journal of Visual Communication and Image Representation, 2019,60:319-327.
[8]	SHAHDOOSTI H R, TABATABAEI Z . MRI and PET/SPECT Image Fusion at Feature Level Using Ant Colony Based Segmentation[J]. Biomedical Signal Processing and Control, 2019,47:63-74.
[9]	TALREJA V, VALENTI M C, NASRABADI N M . Multibiometric Secure System Based on Deep Learning[C]//Proceedings of the 2017 IEEE Global Conference on Signal and Information Processing. Piscataway: IEEE, 2017: 298-302.
[10]	SIMONYAN K, ZISSERMAN A . Very Deep Convolutional Networks for Large-scale Image Recognition[C]//Proceedings of the 2015 3rd International Conference on Learning Representations. San Diego: International Conference on Learning Representations, 2015: 149801.
[11]	马苗, 焦莉莉 . 结构相似性灰关联在强噪声图像增强中的应用[J]. 西安电子科技大学学报, 2010,37(2):346-352.
	MA Miao, JIAO Lili . Research on Image Enhancement for the Heavy Noised Image Based on Structural Similarity via Gray Relational Analysis[J]. Journal of Xidian University, 2010,37(2):346-352.
[12]	韩泽 . 基于深度学习的多波段图像自适应融合方法研究[D]. 太原: 中北大学, 2018.
[13]	LIU Y, LIU S P, WANG Z F . A General Framework for Image Fusion Based on Multi-scale Transform and Sparse Representation[J]. Information Fusion, 2015,24:147-164.
[14]	MOONON A U, HU J W . Multi-focus Image Fusion Based on NSCT and NSST[J]. Sensing and Imaging, 2015,16(1):1-16.
[15]	MA J Y, YU W, LIANG P W , et al. FusionGAN: a Generative Adversarial Network for Infrared and Visible Image Fusion[J]. Information Fusion, 2019,48:11-26.

图像	状态	AG	SF	PSNR
No.1	有注意力	5.320	9.728	28.049
No.1	无注意力	3.044	6.066	27.643
No.2	有注意力	6.926	12.668	27.241
No.2	无注意力	4.480	8.551	27.458
No.3	有注意力	5.292	9.410	32.575
No.3	无注意力	2.607	5.668	29.948