基于交叉融合编码器的Transformer图像特征提取网络

doi:10.16180/j.cnki.issn1007-7820.2025.09.003

摘要/Abstract

摘要：

针对基于窗口的视觉Transformer易破坏细粒度特征、模型参数量大等问题,文中提出一种基于交叉融合编码器模块的Transformer图像特征提取网络。利用图像通道特征相关一致性剥离特征图来获得两个特征子集,并联两个注意力模块分别进行注意计算以获得局部和全局信息,采用交叉机制融合信息。结合CAT Transformer的窗口间注意力模块设计一种用于特征图通道维度之间的窗口内注意力方式,以避免破坏纹理信息,增强局部特征的表征能力。实验结果表明,所提模型在CIFAR-100数据集使用7.8 MB参数达到79.86%的TOP-1准确率,在ImageNet-1K数据集达到80.7%的准确率,同时利用Grad-CAM(Gradient-weighted Class Activation Mapping)来可视化模型决策过程。

关键词: 计算机视觉, 图像分类, 自注意力机制, 特征提取, 上下文信息, 编码器, 通道特征, 卷积神经网络

Abstract:

In view of the problems that window-based vision Transformer is easy to destroy fine-grained features and large number of model parameters, this study proposes a cross-fusion encoder based Transformer image feature extraction network. Two feature subsets are obtained using image channel feature correlation consistency stripping feature maps. Two attention modules are connected in parallel perform attention calculations respectively to obtain local and global information. A crossover mechanism is adopted to fuse information. Combined with the inter-window attention module of CAT Transformer, an in-window attention mode between channel dimensions of feature graph is designed to avoid destroying texture information and enhance the representation ability of local features. Experimental results show that the proposed model achieves 79.86% TOP-1 accuracy with 7.8 MB parameter on CIFAR-100 data set and 80.7% accuracy on ImageNet-1K data set. Grad-CAM(Gradient-weighted Class Activation Mapping)is also used to visualize the decision-making process.

Key words: computer vision, image classification, self-attention, feature extraction, contextual information, encoder, channel feature, convolutional neural network

中图分类号:

TP391.4文献标识码A

龚宇, 吴鹏. 基于交叉融合编码器的Transformer图像特征提取网络[J]. 电子科技, 2025, 38(9): 20-25.

GONG Yu, WU Peng. Cross-Fusion Encoder-Based Transformer Feature Extraction Network[J]. Electronic Science and Technology, 2025, 38(9): 20-25.

图/表 9

图1

图2

图3

图4

表1

表2

表3

表4

图5

参考文献 26

[1]	Yu J, Qin Z, Wan T. Feature integration analysis of bag of features model for image retrieval[J]. Neurocomputing, 2013, 1(120):355-364.
[2]	Lecun Y, Boser J S. Back propagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(4):541-551.
[3]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 1(30):5998-6008.
[4]	Dosovitskiy A, Lucas Beyer. An image is worth 16×16 words:Transformers for image recognition at scale[C]. Online: The Eighth International Conference on Learning Representations, 2020:158-162.
[5]	Touvron H, Cored M, Douze M, et al. Training data-efficient image transformers and distillation through attention[C]. Vienna: Proceedings of the International Conference on Machine Learning, 2021:10347-10357.
[6]	Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]. Online: Proceedings of the European Conference on Computer Vision, 2020:213-229.
[7]	Wang W H, Xie E Z, Li X, et al. Pyramid vision Transformer:A versatile backbone for dense prediction without convolutions[C]. Montreal: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021:568-578.
[8]	Graham B, El-Nouby A, Touvron H, et al. Levit:A vision transformer in convnet's clothing for faster inference[C]. Montreal: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021:12259-12269.
[9]	Mehta S, Rastegari M. Mobilevit: Light-weight,general-purpose and mobile-friendly vision transformer[EB/OL].(2022-5-22)[2024-01-4]. https://arxiv.org/abs/2110.02178.
[10]	Pan J, Bulat A, Tan F, et al. Edgevits:Competing light-weight CNNs on mobile devices with vision transformers[C]. Tel-Aviv: European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022:294-311.
[11]	Chen X, Zhang S, Song D, et al. Transformer with bidirectional decoder for speech recognition[C]. Shanghai: Proceedings of the Annual Conference of the International Speech Communication Association, 2020:1-12.
[12]	Guo J, Han K, Wu H, et al. CMT:Convolutional neural networks meet vision transformers[C]. Hulun Buir: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022:12175-12185.
[13]	He K, Zhang X, Ren S, et al. Deep residual learning for-image recognition[C]. Las Vegas: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:770-778.
[14]	孙红, 杨晨, 莫光萍. 基于通道特征金字塔的图像分割算法[J]. 电子科技, 2023, 36(12):1007-7082.
	Sun Hong, Yang Chen, Mo Guangping. Research on image segmentation algorithm based on channel feature pyramid[J]. Electronic Science and Technology, 2023, 36(12):1007-7082.
[15]	Liu Z, Lin Y, Cao Y, et al. Swin transformer:Hierarchical vision transformer using shifted windows[C]. Montreal: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021:10012-10022.
[16]	Hassani A, Walton S, Li J, et al. Neighborhood attention-transformer[C]. Paris: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023:6185-6194.
[17]	Lin H, Cheng X, Wu X, et al. Cat:Cross attention in vision transformer[C]. Taipei: IEEE International Conference on Multimedia and Expo, 2022:1-6.
[18]	Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4,inception-resnet and the impact of residual connections on learning[C]. San Francisco: Proceedings of the AAAI Conference on Artificial Intelligence, 2017:1-22.
[19]	Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images[J]. Toronto:University of Toronto, 2012:76-99.
[20]	Deng J, Dong W, Socher R, et al. Imagenet:A large-scale hierarchical image database[C]. Miami: IEEE Conference on Computer Vision and Pattern Recognition, 2009:248-255.
[21]	Cubuk E D, Zoph B. Randaugment:Practical automated data augmentation with a reduced search space[C]. Online: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020:702-703.
[22]	Zhong Z, Zheng L, Kang G. Random erasing data augmentation[C]. New York: Proceedings of the AAAI Conference on Artificial Intelligence, 2020:13001-13008.
[23]	Yun S, Han D. Regularization strategy to train strong classifiers with localizable features[C]. Seoul: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019:6023-6032.
[24]	Zhang Z, Zhang H, Zhao L, et al. Aggregating nested transfomers[EB/OL].(2021-10-30)[2024-01-12]. https://arxiv.org/abs/2105.12723.
[25]	Radosavovic I, Kosaraju R P, Girshick R. Designing network design spaces[C]. Online: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020:10428-10436.
[26]	He K, Zhang X, Ren S. Deep residual learning for image recognition[C]. New York: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:770-778.

数据集	任务	训练集	测试集	类别
CIFAR-100	图像分类	5.00×10⁴	1.0×10⁴	100
ImageNet-1K	图像分类	1.28×10⁶	5.0×10⁴	1 000

超参数	ImageNet-1K	CIFAR-100
Epochs/Optimizer	300/AdamW	300/AdamW
Batch Size/LR	1 024/0.01	512/0.05
Warmup epoch	20	20
Color-Jitter/Mixup	0.4/0.8	×/×

架构	模型	参数量/10⁷	精度/%
Transformer	Deit-T	8.67	67.52
	PVT-T	2.45	69.62
	Swin-T	2.83	78.07
	NesT-T	1.70	79.69
	本文	0.78	79.86

架构	模型	FLOPs/10⁹	精度/%
CNN	ResNet-50	4.1	76.2
	ResNet-101	7.9	77.4
	X50-32x4d	4.3	77.9
	RegNetY-4G	4.0	80.0
Transformer	ViT-B	55.4	77.9
	Deit-T	4.6	79.7
	PVT-T	3.8	79.8
	Swin-T	4.5	81.3
	本文	2.1	80.7