基于深度学习的红外与可见光图像匹配

doi:10.16180/j.cnki.issn1007-7820.2025.09.001

摘要/Abstract

摘要：

基于卷积神经网络(Convolutional Neural Network, CNN)的图像相似度检测算法对图像特征的表达能力较差,且只适用于单一特定任务,易出现过拟合风险。针对该问题,文中提出一种基于深度学习的红外与可见光图片匹配方法(Deep Learning-based Image Matching of Infrared and Visible Image, DLIVM)。采用批通道归一化(Batch Channel Normalization, BCN)、注意力机制、度量学习和Frobenius范数等技术提升图像匹配性能和泛化能力。使用将批归一化层(Batch Normalization, BN)改进为BCN的ResNet-50(Residual Neural Network-50)网络作为主干网络提取图片特征,同时在残差单元内部加入注意力机制。使用二元交叉熵损失和度量学习相结合的方法构建目标函数,提升特征表示判别能力。使用Frobenius范数对模型参数进行正则化,以防止过拟合。结果表明,在3个广泛使用的红外与可见光数据集上,相较于对比方法,DLIVM方法的准确率分别提高了3.30%、0.86%、2.00%、7.50%、1.50%、0.69%。

关键词: 图像匹配, 卷积神经网络, 批通道归一化, 异源图像匹配, 注意力机制, 度量学习, 深度学习, 模态不变性

Abstract:

The image similarity detection algorithm based on CNN(Convolutional Neural Network) has poor ability to express image features and is only suitable for a single specific task, which is prone to overfitting risk. In this study, a method of DLIVM(Deep Learning-based Image Matching of Infrared and Visible Image) is proposed. This method uses BCN(Batch Channel Normalization), attention mechanism, metric learning and Frobenius norm to improve image matching performance and generalization ability. The ResNet-50(Residual Neural Network-50)network, which modified the BN(Batch Normalization) layer to BCN, is used as the backbone network to extract image features, and the attention mechanism is added inside the residual unit. The objective function is constructed by combining binary cross entropy loss and metric learning to improve the distinguishing ability of feature representation. The model parameters are regularized using the Frobenius norm to prevent overfitting. The results show that on three widely used infrared and visible data sets, the accuracy of DLIVM method is improved by 3.30%, 0.86%, 2.00%, 7.50%, 1.50% and 0.69%, respectively,when compared with the comparison method.

Key words: image matching, convolutional neural network, batch channel normalization, heterologous image matching, attention mechanism, metric learning, deep learning, modality invariance

中图分类号:

TP391

熊子恒, 张轩雄. 基于深度学习的红外与可见光图像匹配[J]. 电子科技, 2025, 38(9): 1-8.

XIONG Ziheng, ZHANG Xuanxiong. Deep Learning-Based Image Matching of Infrared and Visible Image[J]. Electronic Science and Technology, 2025, 38(9): 1-8.

图/表 12

图1

图2

图3

图4

图5

图6

图7

图8

表1

表2

表3

图9

参考文献 24

[1]	刘海桥, 刘萌, 龚子超, 等. 基于深度学习的图像匹配方法综述[J]. 航空学报, 2024, 45(3):28796-28810. doi: 10.7527/S1000-6893.2023.28796
	Liu Haiqiao, Liu Meng, Gong Zichao, et al. A review of image matching methods based on deep learning[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(3):28796-28810. doi: 10.7527/S1000-6893.2023.28796
[2]	Wei Y, Guo H K, Lu J W, et al. Iterative feature matching for self-supervised indoor depth estimation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(6):3839-3852.
[3]	Fu Y J, Zhang P J, Liu B X, et al. Learning to reduce scale differences for large-scale invariant image matching[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(3):1335-1348.
[4]	Jin Y, Mishkin D, Mishchuk A, et al. Image matching across wide baselines:From paper to practice[J]. International Journal of Computer Vision, 2021, 129(2):517-547.
[5]	Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2):91-110.
[6]	Rublee E, Rabaud V, Konolige K, et al. ORB:An efficient alternative to SIFT or SURF[C]. Barcelona: International Conference on Computer Vision, 2011:2564-2571.
[7]	廉蔺, 李国辉, 张军, 等. 基于边缘最优映射的红外和可见光图像自动配准算法[J]. 自动化学报, 2012, 38(4):570-581.
	Lian Lin, Li Guohui, Zhang Jun, et al. An automatic registration algorithm of infrared and visible image based on optimal mapping of edges[J]. Acta Automatica Sinica, 2012, 38(4):570-581.
[8]	闫超, 孙占全, 田恩刚, 等. 基于深度学习的医学图像分割技术研究进展[J]. 电子科技, 2021, 34(2):7-11.
	Yan Chao, Sun Zhanquan, Tian Engang, et al. Research progress of medical image segmentation based on deep learning[J]. Electronic Science and Technology, 2021, 34(2):7-11.
[9]	严镕宇, 李伟, 陈玉明, 等. 一种基于孪生网络的图片匹配算法[J]. 南京大学学报(自然科学), 2023, 59(5):770-776.
	Yan Rongyu, Li Wei, Chen Yuming, et al. A siamese network-based image matching algorithm[J]. Journal of Nanjing University(Natural Science), 2023, 59(5):770-776.
[10]	Hartmann W, Galliani S, Havlena M, et al. Learned multi-patch similarity[C]. Venice: IEEE International Conference on Computer Vision, 2017:1595-1603.
[11]	Huang H, Kalogerakis E, Chaudhuri S, et al. Learning local shape descriptors from part correspondences with multi view convolutional networks[J]. ACM Transactions on Graphics, 2018, 37(1):5-6.
[12]	Schroff F, Kalenichenko D, Philbin J. Facenet:A unified embedding for face recognition and clustering[C]. Boston: IEEE Conference on Computer Vision and Pattern Recognition, 2015:815-823.
[13]	Hughes L H, Schmitt M, Mou L, et al. Identifying corresponding patches in SAR and optical images with a pseudo-siamese CNN[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15(5):784-788.
[14]	Stollenga M F, Masci J, Gomez F, et al. Deep networks with internal selective attention through feedback connections[C]. Montreal: Advances in Neural Information Processing Systems, 2014:3545-3553.
[15]	王文标, 时启衡, 郝友维. 基于改进SqueezeNet的火焰识别算法[J]. 中国安全生产科学技术, 2024, 20(8):19-26.
	Wang Wenbiao, Shi Qiheng, Hao Youwei. Flame recognition algorithm based on improved SqueezeNet[J]. Journal of Safety Science and Technology, 2024, 20(8):19-26.
[16]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]. Las Vegas: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016:770-778.
[17]	Wan G, Fang H B, Wang D C, et al. Ceramic tile surface defect detection based on deep learning[J]. Ceramics International, 2022, 48(8):11085-11093.
[18]	Razakarivony S, Jurie F. Vehicle detection in aerial imagery: A small target detection benchmark[J]. Journal of Visual Communication and Image Representation, 2016, 34(1):187-203.
[19]	Brown M, Süsstrunk S. Multi-spectral SIFT for scene category recognition[C]. Colorado Springs: IEEE Conference on Computer Vision and Pattern Recognition, 2011:177-184.
[20]	邹融平, 朱斌, 王晨阳, 等. 基于残差伪孪生卷积互相关网络的异源遥感图像匹配算法[J]. 激光与光电子学进展, 2022, 59(12):458-466.
	Zou Rongping, Zhu Bin, Wang Chenyang, et al. Heterogeneous remote sensing image matching algorithm based on residual pseudo-siamese convolutional cross-correlation network[J]. Lasers and Optoelectronics Progress, 2022, 59(12):458-466.
[21]	Li H, Wei P, Hu P. AVN:An adversarial variation network model for handwritten signature verification[J]. IEEE Transactions on Multimedia, 2021, 24(1):594-608.
[22]	Jain A, Singh S K, Singh K P. Multi-task learning using GNet features and SVM classifier for signature identification[J]. IET Biometrics, 2021, 37(2):117-126.
[23]	Ruiz V, Linares I, Sanchez A, et al. Offline handwritten signature verification using compositional synthetic generation of signatures and siamese neural networks[J]. Neurocomputing, 2020, 37(4):30-41.
[24]	Xiao W H, Ding Y T. A two-stage siamese network model for offline handwritten signature verification[J]. Symmetry, 2022, 14(6):1216-1217.

方法	FRR/%	FAR/%	ACC/%
Pseudo-Siamese Network	1.8	1.5	90.1
DLIVM	0.9	1.3	93.1

方法	FRR/%	FAR/%	ACC/%
RPSCCNet^[20]	0.62	0	92.6
DLIVM	0.51	0	93.4

方法	FRR/%	FAR/%	ACC/%
AVN^[21]	11.52	5.73	91.63
GNet_SVM^[22]	17.68	12.24	86.95
Offline_Siamese^[23]	6.58	4.36	92.10
Two-Stage Siamese^[24]	3.48	7.20	92.86
DLIVM	0.00	0.00	93.50