一种高光谱与LiDAR特征耦合的融合分类网络

doi:10.19665/j.issn1001-2400.20240702

摘要/Abstract

摘要：

高光谱图像数据富含丰富的光谱信息,而LiDAR数据可以提供详细的高程信息。在遥感处理领域,通常将二者融合以提高解译精度。然而,在进行不同数据源之间的信息交互时,现有方法没有充分发挥出多源数据融合的优势。因此,笔者针对两种数据各自的优势设计了一种结合卷积网络与Transformer的双分支融合分类算法。在特征提取阶段,设计了跨模态特征耦合模块,通过通道特征交互和空间特征交互操作,提高了主干网络提取特征的一致性,增强了数据间的互补性与特征之间的语义相关性;在特征融合阶段,设计了双边注意力特征融合模块,该模块采用交叉注意力机制对高光谱数据和LiDAR数据进行双向特征融合操作,提升数据间的互补性、减少冗余信息,确保优化后的特征能高效地融合并输入分类器中,从而显著提升了分类网络的准确性与鲁棒性。实验结果表明,与现有的融合分类算法相比,笔者设计的算法在Houston2013和TRENTO数据集上展示出了较为先进的结果,平均分类精度分别提高了1.43%和4.81%,说明本算法可以显著增强网络对高光谱图像空间和光谱特征的辨识能力,提高融合分类准确性。

关键词: 高光谱成像, 激光雷达, 深度学习, 数据融合, 分类

Abstract:

Hyperspectral image data are rich in spectral information,while LiDAR data can provide detailed elevation information.In the field of remote sensing processing,the two are usually fused to improve the interpretation accuracy.However,when performing information interaction between different data sources,existing methods do not fully exploit the advantages of multi-source data fusion.Therefore,this paper presents a dual-branch fusion classification algorithm combining convolutional networks and Transformers,leveraging the advantages of both types of data.In the feature extraction stage,a cross-modal feature coupling module is designed,which improves the consistency of feature extraction by the backbone network through channel feature interaction and spatial feature interaction,enhancing the complementarity between data and the semantic relevance of features.In the feature fusion stage,a bilateral attention feature fusion module is designed which adopts a cross-attention mechanism to perform bidirectional feature fusion operations on hyperspectral data and LiDAR data,enhancing complementarity between data,reducing redundant information,and ensuring that the optimized features can be efficiently fused and input into the classifier so as to significantly improve the accuracy and robustness of the classification network.Experimental results show that compared to existing fusion classification algorithms,the algorithm designed in this paper demonstrates more advanced results on the Houston2013 and TRENTO datasets,with average classification accuracy improvements of 1.43% and 4.81% respectively,indicating that this algorithm can significantly enhance the network’s ability to distinguish spatial and spectral features of hyperspectral images,and improve the fusion classification accuracy.

Key words: hyperspectral imaging, LiDAR, deep learning, data fusion, classification

中图分类号:

TP753

徐海涛, 刘玉哲, 闫欣怡, 李娇娇, 薛长斌. 一种高光谱与LiDAR特征耦合的融合分类网络[J]. 西安电子科技大学学报, 2024, 51(6): 73-83.

XU Haitao, LIU Yuzhe, YAN Xinyi, LI Jiaojiao, XUE Changbin. Fusion classification network for hyperspectral and LiDAR eature coupling modeling[J]. Journal of Xidian University, 2024, 51(6): 73-83.

图/表 10

图1

图2

图3

表1

表2

图4

图5

表3

表4

图6

参考文献 26

[1]	张良培, 沈焕锋. 遥感数据融合的进展与前瞻[J]. 遥感学报, 2016, 20(5):1050-1061.
	ZHANG Liangpei, SHEN Huanfeng. Progress andFuture of Remote Sensing Data Fusion[J]. National Remote Sensing Bulletin, 2016, 20(5):1050-1061.
[2]	赵伍迪, 李山山, 李安. 结合深度学习的高光谱与多源遥感数据融合分类[J]. 遥感学报, 2021, 25(7):1489-1502.
	ZHAO Wudi, LI Shanshan, LI An, et al. DeepFusion of Hyperspectral Images and Multi-Source Remote Sensing Data for Classification with Convolutional Neural Network[J]. National Remote Sensing Bulletin, 2021, 25(7):1489-1502.
[3]	WANG J, LI W, GAO Y, et al. Hyperspectral and SAR Image Classification via Multiscale Interactive Fusion Network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(12):10823-10837.
[4]	ZHU X X, TUIA D, MOU L, et al. Deep Learning in Remote Sensing:A Comprehensive Review and List of Resources[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4):8-36.
[5]	BALL J E, ANDERSON D T, CHAN C S. Comprehensive Survey of Deep Learning in Remote Sensing:Theories,Tools,and Challenges for the Community[J]. Journal of Applied Remote Sensing, 2017, 11(4):042609.
[6]	CHENG G, XIE X, HAN J, et al. Remote Sensing Image Scene Classification Meets Deep Learning:Challenges,Methods,Benchmarks,and Opportunities[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:3735-3756.
[7]	JIN Q, LUO J, LI Y, et al. Scene Classification of Remote Sensing Images Based on Improved HrNet[C]// 2023 3rd International Conference on Electronic Information Engineering and Computer Science(EIECS). Piscataway:IEEE, 2023:952-958.
[8]	ZHANG Z, MI X, YANG J, et al. Remote Sensing Image Scene Classification in Hybrid Classical-Quantum Transferring CNN with Small Samples[J]. Sensors, 2023, 23(18):8010.
[9]	王梨名, 祁昆仑, 杨超. 弱监督尺度自适应增强的高分辨率遥感影像场景分类[J]. 遥感学报, 2023, 27(12):2815-2830.
	WANG Liming, QI Kunlun, YANG Chao, et al. Weakly Supervised Scale Adaptation Data Augmentation for Scene Classification of High-Resolution Remote Sensing Images[J]. National Remote Sensing Bulletin, 2023, 27(12):2815-2830.
[10]	KROUPI E, KESA M, NAVARRO-SÁNCHEZ V D, et al. Deep Convolutional Neural Networks for Land-Cover Classification with Sentinel-2 Images[J]. Journal of Applied Remote Sensing, 2019, 13(2):024525.
[11]	He M, Li B, Chen H. Multi-scale 3D Deep Convolutional Neural Network for Hyperspectral Image Classification[C]// 2017 IEEE International Conference on Image Processing (ICIP). Piscataway:IEEE, 2017:3904-3908.
[12]	YANG X, YE Y, LI X, et al. Hyperspectral Image Classification with Deep Learning Models[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9):5408-5423.
[13]	薛朝辉, 张瑜娟. 基于卷积核哈希学习的高光谱图像分类[J]. 遥感学报, 2022, 26(4):722-738.
	XUE Zhaohui, ZHANG Yujuan. Supervised Hashing with RBF Kernel and Convolution for Hyperspectral Image Classification[J]. National Remote Sensing Bulletin, 2022, 26(4):722-738.
[14]	熊敬伟, 潘继飞, 毕大平. 面向雷达行为识别的多尺度卷积注意力网络[J]. 西安电子科技大学学报, 2023, 50(6):62-74.
	XIONG Jingwei, PAN Jifei, BI Daping, et al. Multi-Scale Convolutional Attention Network for Radar Behavior Recognition[J]. Journal of Xidian University, 2023, 50(6):62-74.
[15]	QING Y, LIU W, FENG L, et al. Improved Transformer Net for Hyperspectral Image Classification[J]. Remote Sensing, 2021, 13(11):2216.
[16]	AYAS S, TUNC-GORMUS E. SpectralSWIN:A Spectral-SWIN Transformer Network for Hyperspectral Image Classification[J]. International Journal of Remote Sensing, 2022, 43(11):4025-4044.
[17]	CHEN Y, LI C, GHAMISI P, et al. Deep Fusion of Remote Sensing Data for Accurate Classification[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(8):1253-1257.
[18]	LI H, GHAMISI P, SOERGEL U, et al. Hyperspectral and LiDAR Fusion Using Deep Three-Stream Convolutional Neural Networks[J]. Remote Sensing, 2018, 10(10):1649.
[19]	XU X, LI W, RAN Q, et al. Multisource Remote Sensing Data Classification Based on Convolutional Neural Network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(2):937-949.
[20]	HANG R, LI Z, GHAMISI P, et al. Classification of Hyperspectral and LiDAR Data Using Coupled CNNs[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(7):4939-4950.
[21]	ZHAO X, TAO R, LI W, et al. Joint Classification of Hyperspectral and LiDAR Data Using Hierarchical Random Walk and Deep CNN Architecture[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(10):7355-7370.
[22]	MOHLA S, PANDE S, BANERJEE B, et al. Fusatnet:Dual Attention Based Spectrospatial Multimodal Fusion Network for Hyperspectral and Lidar Classification[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE, 2020:92-93.
[23]	WANG X, FENG Y, SONG R, et al. Multi-Attentive Hierarchical Dense Fusion Net for Fusion Classification of Hyperspectral and LiDAR Data[J]. Information Fusion, 2022, 82:1-18.
[24]	SHI X, LIN J, RAO Y, et al. Gated-Cross Aggregation Network for Hyperspectral and LiDAR Data Classification[C]// IGARSS 2023-2023 IEEE International Geoscience and Remote Sensing Symposium. Piscataway:IEEE, 2023:1265-1268.
[25]	HONG D, HAN Z, YAO J, et al. SpectralFormer:Rethinking Hyperspectral Image Classification with Transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60:1-15.
[26]	XUE Z, TAN X, YU X, et al. Deep Hierarchical Vision Transformer for Hyperspectral and LiDAR Data Classification[J]. IEEE Transactions on Image Processing, 2022, 31:3095-3110. doi: 10.1109/TIP.2022.3162964 pmid: 35404817

类别(Houston2013)		样本数		类别(TRENTO)		样本数
编号	名称	训练	测试	编号	名称	训练	测试
1	健康草地	198	1053	1	树木	129	3905
2	倒伏草地	190	1064	2	建筑物	125	2778
3	人造草皮	192	505	3	地面	105	374
4	树木	188	1056	4	森林	154	8969
5	土壤	186	1056	5	葡萄园	184	10317
6	水源	182	143	6	道路	122	3525
7	住宅区	196	1072
8	商业区	191	1053
9	道路	193	1059
10	高速路	191	1036
11	铁路	181	1054
12	停车场1	192	1041
13	停车场2	184	285
14	网球场	181	247
15	跑道	187	473
总计		2832	12197	总计		819	29595

类别	CNN-PPF	Two-Branch	HRWN	ViT	SpectralFormer	DHViT	Ours
树木	97.13%	89.29%	91.45%	87.35%	96.08%	94.65%	99.54%
建筑物	92.12%	91.22%	97.83%	81.21%	95.86%	98.42%	99.35%
地面	98.93%	83.72%	92.48%	96.79%	95.99%	81.55%	99.73%
森林	99.10%	98.08%	98.31%	97.42%	97.99%	99.96%	99.78%
葡萄园	96.74%	100.00%	99.86%	74.66%	95.25%	99.75%	98.35%
道路	68.32%	87.27%	83.08%	69.95%	57.76%	86.14%	93.84%
AA	94.14%	95.55%	96.19%	84.57%	92.37%	96.62%	98.48%
OA	92.05%	91.61%	93.84%	83.70%	89.82%	93.61%	98.65%
Kappa	0.9216	0.9403	0.9419	0.7844	0.8982	0.9547	0.9819

类别	CNN-PPF	Two-Branch	HRWN	ViT	SpectralFormer	DHViT	文中算法
健康草地	83.00%	83.10%	85.31%	82.72%	81.86%	81.58%	95.73%
倒伏草地	84.12%	84.87%	83.79%	80.45%	100.00%	78.48%	87.78%
人造草皮	100.00%	100.00%	99.05%	99.60%	95.25%	74.26%	99.80%
树木	88.54%	92.14%	92.30%	92.42%	96.12%	90.25%	92.33%
土壤	100.00%	97.73%	100.00%	97.73%	99.53%	99.72%	99.15%
水源	97.20%	68.53%	97.28%	95.80%	94.41%	87.41%	86.01%
住宅区	83.40%	87.33%	89.33%	74.44%	83.12%	84.70%	95.71%
商业区	46.25%	70.75%	93.74%	42.55%	76.73%	95.35%	91.36%
道路	84.04%±	84.51%	88.66%	65.25%	79.32%	77.15%	88.86%
高速路	56.37%	62.64%	86.17%	50.77%%	78.86%	42.47%	81.08%
铁路	80.08%	76.47%	92.75%	71.44%	88.71%	79.60%	86.62%
停车场1	87.42%	91.26%	96.47%	56.00%	87.32%	96.45%	85.49%
停车场2	82.81%	8.12%	91.93%	64.21%	72.63%	79.30%	88.42%
网球场	100.00%	100.00%	100.00%	100.00%	100.00%	99.60%	90.28%
跑道	98.94%	98.93%	100.00%	98.52%	100.00%	87.53%	100.00%
AA	84.81%	82.70%	90.47%	78.13%	88.91%	83.59%	91.24%
OA	81.69%	80.42%	89.67%	74.36%	88.01%	82.79%	91.10%
Kappa	0.803	0.8124	0.8828	0.7243	0.8699	0.8137	0.9033

数据集	指标	基础网络	基础网络+CMFM	基础网络+BAFM	文中算法
Houston	OA	85.27%	87.61%	87.91%	91.10%
	AA	87.78%	89.16%	88.41%	91.24%
	Kappa	0.8408	0.8657	0.8682	0.9033
TRENTO	OA	97.33%	98.12%	98.44%	98.65%
	AA	96.43%	97.35%	98.23%	98.48%
	Kappa	0.9644	0.9769	0.9792	0.9819