一种自适应权重学习的轻量超分辨率重建网络

doi:10.19665/j.issn1001-2400.2021.05.003

摘要/Abstract

摘要：

近年来,基于深度卷积神经网络的单幅图像超分辨率重建方法取得了令人瞩目的成果。基于像素级注意力网络的图像超分辨率重建方法能够在极小的参数量下获得良好的重建性能,是目前最先进的轻量化超分辨率重建方法之一。但是,受到各模块参数量的限制,像素级注意力网络训练缓慢和收敛条件苛刻的问题变得日益突出。针对这些问题,提出了一种基于自适应权重学习的轻量化超分辨率重建网络。该网络使用多个自适应权重模块组成非线性映射网络,每个模块能够提取到不同层级的特征信息,在每个自适应权重模块中,利用注意力分支和无注意力分支分别获取相应信息,再通过自适应权重融合分支进行整合。使用特定的卷积层拆分和融合两条分支,大幅降低了注意力分支和无注意力分支的参数量,使网络在参数量与性能之间达到相对平衡。在标准数据集上的实验证明,所提出方法在降低模型参数量的同时,峰值信噪比和结构相似度两种客观质量评价指标均优于同类先进方法,该方法能够重建更准确的纹理细节,得到更好的视觉效果,证明了该方法的有效性。

关键词: 超分辨率重建, 卷积神经网络, 深度学习, 轻量化, 自适应权重

Abstract:

In recent years,the single-image super-resolution (SISR) method using deep convolutional neural networks (CNN) has achieved remarkable results.The Pixel Attention Network(PAN) is one of the most advanced lightweight super-resolution methods,which can lead to a good reconstruction performance with a very small number of parameters.But the PAN is limited by the parameters of each module,resulting in slow model training and strict training conditions.To address these problems,this paper proposes a Lightweight Adaptive Weight learning Network (LAWN) for image super-resolution.The network uses multiple adaptive weight modules to form a non-linear mapping network,with each module extracting different levels of feature information.In each adaptive weight module,the network employs the attention branch and the non-attention branch to extract the corresponding information,and then the adaptive weight fusion branch is employed to integrate these two branches.Splitting and fusing the two branches with a specific convolutional layer greatly reduces the number of parameters of the attention branch and the non-attention branch,which helps the network to achieve a relative balance between the number of parameters and the performance.The quantitative evaluations on benchmark datasets demonstrate that the proposed LAWN reduces the number of model parameters and performs favorably against state-of-the-art methods in terms of both PSNR and SSIM.Experimental results show that this method can reconstruct more accurate texture details.The qualitative evaluations with better visual effects prove the effectiveness of the proposed method.

Key words: super-resolution, convolutional neural networks, deep learning, lightweight, adaptive weight

中图分类号:

TP391

张宇浩,程培涛,张书豪,王秀美. 一种自适应权重学习的轻量超分辨率重建网络[J]. 西安电子科技大学学报, 2021, 48(5): 15-22.

ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network[J]. Journal of Xidian University, 2021, 48(5): 15-22.

图/表 6

图1

图2

表1

图3

图4

图5

参考文献 37

[1]	苏衡, 周杰, 张志浩. 超分辨率图像重建方法综述[J]. 自动化学报, 2013, 39(8):1202-1213.
	SU Heng, ZHOU Jie, ZHANG Zhihao. Survey of Super-Resolution Image Reconstruction Methods[J]. Acta Automatica Sinica, 2013, 39(8):1202-1213.
[2]	WANG Z H, CHEN J, HOI S C H. Deep Learning for Image Super-Rsolution:A Survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 45(10):3365-3387.
[3]	DONG C, LOY C C, HE K M, et al. Image Super-Resolution Using Deep Convolutional Networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307. doi: 10.1109/TPAMI.2015.2439281
[4]	KIM J, LEE J K, LEE K M. Deeply-Recursive Convolutional Network for Image Super-Resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:1637-1645.
[5]	ZHANG Y L, LI K P, LI K, et al. Image Super-Resolution Using Very Deep Residual Channel Attention Networks[C]// Proceedings of the European Conference on Computer Vision (ECCV).Piscataway:IEEE, 2018:286-301.
[6]	KIM J, LEE J K, LEEK M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:1646-1654.
[7]	LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Piscataway:IEEE, 2017:136-144.
[8]	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual Dense Network for Image Super-Resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:2472-2481.
[9]	TAI Y, YANG J, LIU X. Image Super-Resolution Bia Deep Recursive Residual Network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:3147-3155.
[10]	王世平, 毕笃彦, 刘坤, 等. 一种多映射卷积神经网络的超分辨率重建算法[J]. 西安电子科技大学学报, 2018, 45(4):155-160.
	WANG Shiping, BI Duyan, LIU Kun, et al. Multi-Mapping Convolution Neural Network for the Image Super-Resolution Algorithm[J]. Journal of Xidian University, 2018, 45(4):155-160.
[11]	SOH J W, CHO S, CHON I. Meta-Transfer Learning for Zero-Shot Super-Resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2020:3516-3525.
[12]	ZHANG K, GOOL L V, TIMOFTE R. Deep Unfolding Network for Image Super-Resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2020:3217-3226.
[13]	JO Y, KIM S J. Practical Single-Image Super-Resolution Using Look-Up Table[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2021:691-700.
[14]	GU J, DONG C. Interpreting Super-Resolution Networks with Local Attribution Maps[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2021:9199-9208.
[15]	SONG D, WANG Y, CHEN H, et al. Addersr:Towards Energy Efficient Image Super-Resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2021:15648-15657.
[16]	HUI Z, LI J, WANG X, et al. Learning the Non-Differentiable Optimization for Blind Super-Resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2021:2093-2102.
[17]	刘树东, 王晓敏, 张艳. 一种对称残差CNN的图像超分辨率重建方法[J]. 西安电子科技大学学报, 2019, 46(5):15-23.
	LIU Shudong, WANG Xiaomin, ZHANG Yan. Symmetric Residual Convolution Neural Networks for the Image Super-Resolution Reconstruction[J]. Journal of Xidian University, 2019, 46(5):15-23.
[18]	LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:624-632.
[19]	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:4700-4708.
[20]	HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:7132-7141.
[21]	DAI T, CAI J, ZHANG Y, et al. Second-Order Attention Network for SingleImage Super-Resolution[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019:11065-11074.
[22]	ZHANG Y, LI K, LI K, et al. Residual Non-local Attention Networks for Image Restoration[EB/OL]. [2019-03-24]. https://arxiv.org/pdf/1903.10082.pdf.
[23]	WANG X, GIRSHICK R, GUPTA A, et al. Non-Local Neural Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:7794-7803.
[24]	NIU B, WEN W, REN W, et al. Single Image Super-Resolution via a Holistic Attention Network[C]// European Conference on Computer Vision.Berlin:Springer, 2020:191-207.
[25]	AHN N, KANG B, SOHN K A. Fast,Accurate,and Lightweight Super-Resolution with Cascading Residual Network[C]// Proceedings of the European Conference on Computer Vision.Berlin:Springer, 2018:252-268.
[26]	HUI Z, WANG X, GAOG X. Fast and Accurate Single Image Super-Resolution via Information Distillation Network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2018:723-731.
[27]	HUI Z, GAO X, YANG Y, et al. Lightweight Image Super-Resolution with Information Multi-Distillation Network[C]// Proceedings of the 27th ACM International Conference on Multimedia.New York:ACM, 2019:2024-2032.
[28]	TAI Y, YANG J, LIU X, et al. Memnet:A Persistent Memory Network for Image Restoration[C]// Proceedings of the IEEE International Conference on Computer Vision.Piscataway:IEEE, 2017:4539-4547.
[29]	ZHAO H, KONG X, HE J, et al. Efficient Image Super-Resolution Using Pixel Attention[C]// European Conference on Computer Vision.Berlin:Springer, 2020:56-72.
[30]	CHEN H, GU J, ZHANG Z. Attention in Attention Network for Image Super-Resolution[EB/OL]. [2021-04-19] et al. https://arxiv.org/pdf/2104.09497.pdf.
[31]	CHEN Y, DAI X, LIU M, et al. Dynamic Convolution:Attention over Convolution Kernels[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2020:11030-11039.
[32]	AGUSTSSON E, TIMOFTE R. Ntire 2017 Challenge on Single Image Super-Resolution:Dataset and Study[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Piscataway:IEEE, 2017:126-135.
[33]	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-Complexity Single-Image Super-Resolution Based on Nonnegative Neighbor Embedding[C]// Proceedings of the 23rd British Machine Vision Conference.Surrey:BMVA, 2012: 135.
[34]	ZEYDE R, ELAD M, PROTTER M. On Single Image Scale-Up Using Sparse-Representations[C]// International Conference on Curves and Surfaces.Berlin:Springer, 2010:711-730.
[35]	MARTIN D, FOWLKES C, TALS D, et al. A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics[C]// Proceedings Eighth IEEE International Conference on Computer Vision.Piscataway:IEEE, 2001:416-423.
[36]	HUANG J B, SINGH A, AHUJA N. Single Image Super-Resolution from Transformed Self-Exemplars[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2015:5197-5206.
[37]	WANG Z, BOVIK A C, SHEIKHH R, et al. Image Quality Assessment:from Error Visibility to Structural Similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4):600-612. doi: 10.1109/TIP.2003.819861

方法	放大倍数	参数量	Set5 (PSNR/SSIM)	Set14 (PSNR/SSIM)	BSD100 (PSNR/SSIM)	Urban100 (PSNR/SSIM)
DRRN	×2	298×10³	37.74/0.959 1	33.23/0.913 6	32.05/0.897 3	31.23/0.918 8
IDN		553×10³	37.83/0.960 0	33.30/0.914 8	32.08/0.898 5	31.27/0.919 6
CARN		1 592×10³	37.76/0.959 0	33.52/0.916 6	32.09/0.897 8	31.92/0.925 6
IMDN		694×10³	38.00/0.960 5	33.63/0.917 7	32.19/0.899 6	32.17/0.928 3
PAN		261×10³	37.95/0.960 7	33.59/0.917 3	32.15/0.900 1	31.97/0.927 0
LAWN		454×10³	38.03/0.961 0	33.64/0.918 3	32.19/0.900 5	32.17/0.928 4
DRRN	×3	298×10³	34.03/0.924 4	29.96/0.834 9	28.95/0.800 4	27.53/0.837 8
IDN		553×10³	34.11/0.925 3	29.99/0.835 4	28.95/0.801 3	27.42/0.835 9
CARN		1 592×10³	34.29/0.925 5	30.29/0.840 7	29.06/0.803 4	28.06/0.849 3
IMDN		703×10³	34.36/0.927 0	30.32/0.841 7	29.09/0.804 6	28.17/0.851 9
PAN		261×10³	34.31/0.927 0	30.27/0.841 9	29.06/0.805 8	27.99/0.849 3
LAWN		454×10³	34.42/0.927 8	30.35/0.841 8	29.09/0.805 7	28.19/0.852 5
DRRN	×4	298×10³	31.68/0.888 8	28.21/0.772 0	27.38/0.728 4	25.44/0.763 8
IDN		553×10³	31.82/0.890 3	28.25/0.773 0	27.41/0.729 7	25.41/0.763 2
CARN		1 592×10³	32.13/0.893 7	28.60/0.780 6	27.58/0.734 9	26.07/0.783 7
IMDN		715×10³	32.21/0.894 8	28.58/0.781 1	27.56/0.735 3	26.04/0.783 8
PAN		272×10³	31.77/0.891 3	28.39/0.778 7	27.45/0.734 1	25.64/0.772 1
LAWN		465×10³	32.21/0.896 4	28.56/0.781 7	27.56/0.737 3	26.11/0.786 8