改进SwinIR的多特征融合图像超分辨率重建

doi:10.19665/j.issn1001-2400.20240911

摘要/Abstract

摘要：

针对目前基于先进方法SwinIR在图像超分辨率重建过程中,存在对低分辨率图像局部信息建模能力不足导致特征提取不充分使重建图像质量不佳的问题,提出了一种改进SwinIR的多特征融合图像超分辨率重建方法。所提算法在深层特征提取模块部分,首先设计了若干个串联的残差Swin Transformer块(RSTB),利用RSTB的Swin Transformer层进行长距离依赖建模提取图像的高频信息,使用残差连接实现不同级别特征聚合。其次,设计了交替串联的空间注意力模块和通道注意力模块,弥补RSTB局部建模能力的不足,使网络能够捕捉到图像空间与通道维度遗漏的上下文信息,促进边缘细节信息的重建。最后,通过长跳跃连接将浅层特征与深层特征求和进行融合传输到重建模块进行高质量图像重建。实验结果表明:在放大倍数为2、3、4的4个测试集上,所提改进算法相较SwinIR在峰值信噪比和结构相似度上均取得了较好的结果,而且在视觉效果上重建图像的边缘结构和整体轮廓都更加清晰。

关键词: 图像超分辨率重建, Swin Transformer, 空间注意力, 通道注意力, 多特征融合

Abstract:

In the process of image super-resolution reconstruction based on the advanced SwinIR method,there is a problem that the local information modeling ability of low-resolution images is insufficient,resulting in inadequate feature extraction and poor quality of reconstructed images.An improved SwinIR multi-feature fusion image super-resolution reconstruction method is proposed.In the deep feature extraction module,the proposed algorithm first designs several series residual Swin Transformer blocks(RSTB),uses the Swin Transformer layer(STL) of RSTB for long-distance dependent modeling to extract high-frequency image information,and uses residual connections to achieve different levels of feature aggregation.Second,an alternating series spatial attention module and channel attention module(SA-CA) are designed to make up for the lack of local modeling ability of RSTB,so that the network can capture the missing context information on image space and channel dimension,and promote the reconstruction of edge details.Finally,the summation of shallow features and deep features is fused and transmitted to the reconstruction module for high-quality image reconstruction through along jump connection.Experimental results show that in the four test sets with magnifications of 2,3,and 4,the proposed improved algorithm achieves better results than SwinIR in terms of the peak signal-to-noise ratio and structural similarity,and the edge structure and overall contour of the reconstructed image are clearer in terms of visual effects.

Key words: image super-resolution reconstruction, swin transformer, spatial attention, channel attention, multi-feature fusion

中图分类号:

TP391

王进花, 魏婷, 曹洁, 陈莉. 改进SwinIR的多特征融合图像超分辨率重建[J]. 西安电子科技大学学报, 2024, 51(6): 171-181.

WANG Jinhua, WEI Ting, CAO Jie, CHEN Li. Improved SwinIR for multi-feature fusion image super-resolution reconstruction[J]. Journal of Xidian University, 2024, 51(6): 171-181.

图/表 13

图1

图2

图3

图4

图5

表1

表2

表3

图6

图7

图8

表4

表5

参考文献 45

[1]	JIANG J J, WANG C Y, LIU X M, et al. Deep Learning-based Face Super-resolution:A Survey[J]. ACM Computing Surveys, 2023, 55(1).
[2]	詹克羽, 孙岳, 李颖. 一种多尺度三维卷积的视频超分辨率方法[J]. 西安电子科技大学学报, 2021, 48(5):8-14.
	ZHAN Keyu, SUN Yue, LI Ying. A Multi-scale 3D Convolution Video Super Resolution Method[J]. Journal of Xidian University, 2021, 48(5):8-14.
[3]	REN S, GUO K H, ZHOU X K, et al. Medical Image Super-Resolution Based on Semantic Perception Transfer Learning[J]. IEEE-Acm Transactions on Computational Biology and Bioinformatics, 2023, 20(4):2598-2609.
[4]	须颖, 刘帅, 邵萌, 等. 一种多尺度GAN的低剂量CT超分辨率重建方法[J]. 西安电子科技大学学报, 2022, 49(2):228-236.
	XU Ying, LIU Shuai, SHAO Meng, et al. A Multi-scale,Low-dose CT Super-resolution Reconstruction Method for GAN[J]. Journal of Xidian University, 2022, 49(2):228-236.
[5]	李嫣, 任文琦, 张长青, 等. 基于真实退化估计与高频引导的内窥镜图像超分辨率重建[J]. 自动化学报, 2024, 50(2):334-347.
	LI Yan, REN Wenqi, ZHANG Changqing, et al. Super-resolution Reconstruction of Endoscopic Images Based on True Degradation Estimation and High-frequency Guidance[J]. Acta Automatica Sinica, 2024, 50(2):334-347.
[6]	HAN Z T, HUANG W H. Arbitrary Scale Super-resolution Diffusion Model for Brain MRI Images[J]. Computers in Biology and Medicine, 2024,170.
[7]	TSUTSUMI M, TAKAHASHI T, KOBAYASHI K, et al. Fluorescence Radial Fluctuation Enables Two-photon Super-resolution Microscopy[J]. Frontiers in Cellular Neuroscience, 2023,17.
[8]	XIE B K, LIU S B, LI L. Large-scale Microscope with Improved Resolution Using SRGAN[J]. Optics and Laser Technology, 2024,179.
[9]	ZHANG M, XIN J, ZHANG J, et al. Curvature Consistent Network for Microscope Chip Image Super-Resolution[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(12):10538-10551.
[10]	CHEN J, HUANG S X, CHAN K F, et al. A Hybrid Lens to Realize Electrical Real-Time Super-Resolution Imaging[J]. Laser & Photonics Reviews, 2024.
[11]	WANG X, SUN L J, CHEHRI A, et al. A Review of GAN-Based Super-Resolution Reconstruction for Optical Remote Sensing Images[J]. Remote Sensing, 2023, 15(20).
[12]	时文俊, 郭从洲, 童晓冲, 等. 辐射保真的红外遥感图像超分辨率重建[J]. 西安电子科技大学学报, 2019, 46(2):107-113.
	SHI Wenjun, GUO Congzhou, TONG Xiaochong, et al. Super Resolution Reconstruction of Infrared Remote Sensing Image Based on Radiation Fidelity[J]. Journal of Xidian University, 2019, 46(2):107-113.
[13]	李明锴, 徐其志. 分级监督范式指导下的遥感图像超分辨率方法[J]. 遥感学报, 2024, 28(7):1746-1759.
	LI Ming Kai, XU Qizhi. Super Resolution Method of Remote Sensing Image Guided by Hierarchical Supervision Paradigm[J]. Journal of Remote Sensing, 2024, 28(7):1746-1759.
[14]	ESTEVA A, CHOU K, YEUNG S, et al. Deep Learning-enabled Medical Computer Vision[J]. Npj Digital Medicine, 2021, 4(1):1-9.
[15]	DONG C, LOY C C, HE K, et al. Image Super-Resolution Using Deep Convolutional Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307. doi: 10.1109/TPAMI.2015.2439281 pmid: 26761735
[16]	DONG C, LOY C C, TANG X O. Accelerating the Super-Resolution Convolutional Neural Network[Z]. Computer VisionI-ECCV 2016,PT II. 2016:391-407.
[17]	KIM J, LEE J K, LEE K M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks[C]; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:1646-1654.
[18]	TAI Y, YANG J, LIU X. Image Super-Resolution via Deep Recursive Residual Network[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017:2790-2798.
[19]	LAI W S, HUANG J B, AHUJA N, et al. Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11):2599-2613.
[20]	ZHANG Y L, LI K P, LI K, et al. Image Super-Resolution Using Very Deep Residual Channel Attention Networks[C]. Computer Vision-ECCV 2018, PT VII.2018:294-310.
[21]	XU W, CHEN R W, HUANG B, et al. Enhanced Context Attention Network for Image Super Resolution[J]. IEEE Sensors Journal, 2021, 21(10):11665-11673.
[22]	YU B, LEI B, GUO J Y, et al. Remote Sensing Image Super-Resolution via Residual-Dense Hybrid Attention Network[J]. Remote Sensing, 2022, 14(22).
[23]	QIN J, CHEN L, JEON S, et al. Progressive Interaction-Learning Network for Lightweight Single-Image Super-Resolution in Industrial Applications[J]. IEEE Transactions on Industrial Informatics, 2023, 19(2):2183-2191.
[24]	PARK K, SOH J W, CHO N I. A Dynamic Residual Self-Attention Network for Lightweight Single Image Super-Resolution[J]. IEEE Transactions on Multimedia, 2023, 25:907-918.
[25]	CHEN H T, WANG Y H, GUO T Y, et al. Pre-Trained Image Processing Transformer[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Piscataway:IEEE, 2021:12294-12305.
[26]	GUO C L, YAN Q X, ANWAR S, et al. Image Dehazing Transformer with Transmission-Aware 3D Position Embedding[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR)Piscataway: IEEE, 2022:5802-5810.
[27]	LIANG J Y, CAO J Z, SUN G L, et al. SwinIR:Image Restoration Using Swin Transformer[C]. 2021 IEEE/CVF International Conference on Computer Vision Workshops(ICCV). Piscataway:IEEE, 2021:1833-1844.
[28]	FANG J S, LIN H J, CHEN X Y, et al. A Hybrid Network of CNN and Transformer for Lightweight Image Super-Resolution[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition WorkshopsPiscataway: IEEE, 2022:1102-1111.
[29]	CHOI H, LEE J, YANG J, et al. N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution[Z]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR). Piscataway:IEEE, 2023:2071-2081.
[30]	ZHOU Y P, LI Z, GUO C L, et al. SRFormer:Permuted Self-Attention for Single Image Super-Resolution[C]. 2023 IEEE/CVF International Conference on Computer Vision Workshops(ICCV). Piscataway:IEEE, 2023:12734-12745.
[31]	ROSSI L, BERNUZZI V, FONTANINI T, et al. Swin2-MoSE:A New Single Image Super-Resolution Model for Remote Sensing[J]. arXiv preprint arXiv:240418924, 2024.
[32]	TIMOFTE R, AGUSTSSON E, VAN GOOL L, et al. Ntire 2017 Challenge on Single Image Super-resolution:Methods and Results[J]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE, 2017:1122-1131.
[33]	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding[C]. Proceedings of the British Machine Vision Conference 2012. 2012:1-10.
[34]	ZEYDE R, ELAD M, PROTTER M. On Single Image Scale-up Using Sparse-Representations[C]// International Conference on Curves and Surfaces.[S.1.]:Springer, 2010:711-730.
[35]	MARTIN D, FOWLKES C, TAL D, et al. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics[C]. Proceedings of the IEEE International Conference on Computer Vision(ICCV). Piscataway:IEEE, 2001:416-423.
[36]	HUANG J B, SINGH A, AHUJA N. Single Image Super-resolution from Transformed Self-exemplars[C]. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Piscataway:IEEE, 2015:5197-5206.
[37]	YUAN F, HUANG L F, YAO Y. An Improved PSNR Algorithm for Objective Video Quality Evaluation[C]. Proceedings of the 26th Chinese Control Conference, 2007:376-380.
[38]	ZHOU W, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment:from Error Visibility to Structural Similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4):600-612.
[39]	AHN N, KANG B, SOHN K-A. Fast,Accurate,and Lightweight Super-resolution with Cascading Residual Network[C]. Proceedings of the European Conference on Computer Vision(ECCV). Piscataway:IEEE, 2018:252-268.
[40]	HUI Z, GAO X, YANG Y, et al. Lightweight Image Super-resolution with Information Multi-distillation Network[C]. Proceedings of the 27th ACM International Conference on Multimedia. New York: ACM, 2019:2024-2032.
[41]	LI W, ZHOU K, QI L, et al. Lapar:Linearly-assembled Pixel-adaptive Regression Network for Single Image Super-resolution and Beyond[J]. Advances in Neural Information Processing Systems, 2020, 33:20343-20355.
[42]	LUO X, XIE Y, ZHANG Y, et al. LatticeNet:Towards Lightweight Image Super-Resolution with Lattice Block; In European Conference on Computer Vision(ECCV), 2020:272-289.
[43]	CHEN H, GU J, ZHANG Z. Attention in Attention Network for Image Super-resolution[J]. arXiv preprint arXiv:210409497, 2021.
[44]	LU Z S, LI J C, LIU H, et al. Transformer for Single Image Super-Resolution[Z]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2022:456-465.
[45]	WANG L, LI X, TIAN W, et al. Lightweight Interactive Feature Inference Network for Single-image Super-resolution[J]. Scientific Reports, 2024, 14(1):11589-11601.

算法	Set5		Set14		BSD100		Urban100
算法	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
SRCNN	36.66	0.954 2	32.45	0.906 7	31.36	0.887 9	29.50	0.894 6
FSRCNN	37.05	0.956 0	32.66	0.909 1	31.52	0.892 1	29.88	0.902 0
CARN	37.76	0.959 0	33.52	0.916 6	32.09	0.897 8	31.92	0.925 6
FALSR-A	37.82	0.959 9	33.55	0.916 8	32.11	0.898 7	31.93	0.925 7
IMDN	38.00	0.960 5	33.63	0.917 7	32.19	0.899 6	32.17	0.928 3
LAPAR-A	38.01	0.960 5	33.62	0.918 3	32.19	0.899 9	32.10	0.928 3
LatticeNet	38.15	0.961 0	33.78	0.919 3	32.25	0.900 5	32.43	0.930 2
SwinIR-light	38.14	0.961 1	33.86	0.920 6	32.31	0.901 2	32.76	0.934 0
ESRT	38.03	0.960 0	33.75	0.918 4	32.25	0.900 1	32.58	0.931 8
HNCT	38.08	0.960 8	33.65	0.918 2	32.22	0.900 1	32.22	0.929 4
PILN	38.08	0.960 7	33.72	0.918 1	32.23	0.900 3	32.38	0.930 6
DRSAN	38.13	0.961 0	33.72	0.918 9	32.24	0.900 9	32.41	0.931 2
IFIN	38.00	0.960 6	33.66	0.918 1	32.18	0.899 6	32.14	0.928 4
文中算法	38.21	0.961 3	33.91	0.920 9	32.33	0.901 5	32.80	0.934 2

算法	Set5		Set14		BSD100		Urban100
算法	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
SRCNN	32.75	0.909 0	29.29	0.820 9	28.41	0.786 3	26.24	0.798 9
FSRCNN	33.17	0.914 0	29.40	0.824 1	28.53	0.791 0	26.43	0.808 0
CARN	34.29	0.925 5	30.29	0.840 7	29.06	0.803 4	28.06	0.849 3
IMDN	34.36	0.927 0	30.32	0.841 7	29.09	0.804 6	28.17	0.851 9
LAPAR-A	34.36	0.926 7	30.34	0.842 1	29.11	0.805 4	28.15	0.852 3
LatticeNet	34.53	0.928 1	30.39	0.842 4	29.15	0.805 9	28.33	0.853 8
A2N	34.47	0.927 9	30.44	0.843 7	29.14	0.805 9	28.41	0.857 0
SwinIR-light	34.62	0.928 9	30.54	0.846 3	29.20	0.808 2	28.66	0.862 4
ESRT	34.42	0.926 8	30.43	0.843 3	29.15	0.806 3	28.46	0.857 4
HNCT	34.47	0.927 5	30.44	0.843 9	29.15	0.806 7	28.28	0.855 7
PILN	34.39	0.926 9	30.34	0.841 5	29.08	0.804 8	28.09	0.850 0
DRSAN	34.50	0.927 8	30.39	0.843 7	29.13	0.806 5	28.33	0.856 6
IFIN	34.45	0.927 8	30.47	0.844 2	29.13	0.806 4	28.32	0.856 0
文中算法	34.66	0.929 3	30.59	0.846 6	29.21	0.808 3	28.68	0.862 6

算法	Set5		Set14		BSD100		Urban100
算法	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
SRCNN	30.48	0.862 5	27.50	0.751 0	26.89	0.710 1	24.52	0.722 2
FSRCNN	30.72	0.866 1	27.61	0.754 9	26.97	0.715 0	24.62	0.728 1
CARN	32.13	0.893 7	28.60	0.780 6	27.58	0.734 9	26.07	0.783 7
IMDN	32.21	0.894 8	28.58	0.781 1	27.56	0.735 3	26.04	0.783 8
LAPAR-A	32.15	0.894 4	28.61	0.781 8	27.61	0.736 6	26.14	0.787 1
LatticeNet	32.30	0.896 2	28.68	0.783 0	27.62	0.736 7	26.25	0.787 3
A2N	32.30	0.896 6	28.71	0.784 2	27.61	0.737 4	26.27	0.792 0
SwinIR-light	32.44	0.897 7	28.75	0.785 7	27.69	0.740 6	26.48	0.798 3
ESRT	32.19	0.894 7	28.69	0.783 3	27.69	0.737 9	26.39	0.796 2
HNCT	32.31	0.895 7	28.71	0.783 4	27.63	0.738 1	26.20	0.789 6
PILN	32.22	0.894 9	28.62	0.781 3	27.59	0.736 5	26.19	0.787 8
DRSAN	32.30	0.895 4	28.66	0.783 8	27.61	0.738 1	26.26	0.792 0
IFIN	32.27	0.895 8	28.68	0.783 4	27.62	0.738 1	26.17	0.789 0
文中算法	32.46	0.897 8	28.78	0.785 9	27.70	0.740 8	26.48	0.797 9

模块数	Set5(PSNR)	Set14(PSNR)	BSD100(PSNR)	Urban100(PSNR)
K=2	37.80	33.51	31.98	32.39
K=4	38.21	33.91	32.33	32.80
K=6	38.40	34.17	32.46	33.42
K=8	38.43	34.19	32.48	33.47

放大倍数	模型	参数量/K
	SRCNN	57
	CARN	1 592
	IMDN	715
	LAPAR-A	659
	LatticeNet	777
×4	A2N	1 047
	SwinIR-light	897
	ESRT	751
	HNCT	372
	PILN	600
	IFIN	470
	文中算法	930