一种门控卷积生成对抗网络的图像修复算法

doi:10.19665/j.issn1001-2400.2022.01.023

摘要/Abstract

摘要：

生成对抗网络图像修复算法在填充任意掩码区域时会经常出现错误,原因是其在进行卷积运算时将所有输入像素都视为有效像素。针对该问题,提出一种门控卷积生成对抗网络的图像修复算法,利用门控卷积替换网络残差块中的传统卷积,以有效地学习已知区域与掩码区域之间的关系。算法采用边缘修复加纹理修复的两阶段生成对抗修复网络。首先,用边缘检测算法检测出破损图像中已知区域的结构;然后,将掩码区域的边缘与已知区域的颜色和纹理信息结合起来进行结构修复,再将完整结构与待修复图像一起送入纹理修复网络中进行纹理修复;最终输出得到完整图像。在网络训练过程中,采用谱归一化马尔科夫判别器以改善迭代过程中权重变化缓慢的问题,从而加快收敛速度、提升模型精度。在Places2数据集上的实验结果表明,所提出的算法在修复破损区域形状不一、大小不一的图像时,相较于之前的两阶段修复算法,在峰值信噪比和结构相似性上分别提高了3.8%和3.0%,且主观视觉效果提升明显。

关键词: 图像修复, 门控卷积, 深度学习, 生成对抗网络

Abstract:

The inpainting algorithm for the generative adversarial networks often has the errors when filling arbitrary masked areas,because the convolution operation treats all input pixels as valid pixels.In order to solve this problem,an image inpainting algorithm based on gated convolution is proposed,which uses gated convolution to replace the traditional convolution in the residual block of the network to effectively learn the relationship between the known areas and the masked areas.The algorithm uses a two-stage generation of an adversarial inpainting network with edge repair and texture repair.First,we use the edge detection algorithm to detect the structure of the known area in the damaged image.Next,we combine the edge in the mask area with the color and texture information on the known area to repair the structure.Then,the complete structure is combined with the damaged image and sent to the texture repair network for texture repair.Finally,the complete image is output.In the process of network training,the Spectral Normalized Markovian Discriminator is used to improve the problem of slow weight change in the iterative process,thereby speeding up the convergence speed and improving the accuracy of the model.Experimental results on the Places2 dataset show that the proposed algorithm is different in peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared with the previous two-stage inpainting algorithm when repairing images with different shapes and sizes in damaged areas.The peak signal-to-noise ratio and structural similarity are improved by 3.8% and 3% respectively,and the subjective visual effect is significantly improved.

Key words: image inpainting, gated convolution, deep learning, generative adversarial network

中图分类号:

TP391

高杰,霍智勇. 一种门控卷积生成对抗网络的图像修复算法[J]. 西安电子科技大学学报, 2022, 49(1): 216-224.

GAO Jie,HUO Zhiyong. Algorithmfor image inpainting in generative adversarial networks based on gated convolution[J]. Journal of Xidian University, 2022, 49(1): 216-224.

图/表 8

图1

图2

图3

图4

图5

表1

图6

表2

参考文献 22

[1]	BERTALMIO M, SAPIRO G, CASELLES V, et al. Image Inpainting[C]// Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2000:417-424.
[2]	申海杰, 边倩, 陈晓范, 等. 生成对抗网络用于视频去模糊[J]. 西安电子科技大学学报, 2019, 46(6):112-117.
	SHEN Haijie, BAN Qian, CHEN Xiaofan, et al. Video Deblurring Using the Generative Adversarial Network[J]. Journal of Xidian University, 2019, 46 (6):112-117.
[3]	SRIDEVI G, KUMAR S S. Image Inpainting Based on Fractional-Order Nonlinear Diffusion for Image Reconstruction[J]. Circuits,Systems,and Signal Processing, 2019, 38(8):3802-3817. doi: 10.1007/s00034-019-01029-w
[4]	WEI Y, LIU S. Domain-Based Structure-Aware Image Inpainting[J]. Signal,Image and Video Processing, 2016, 10(5):911-919. doi: 10.1007/s11760-015-0840-y
[5]	ZHANG D, LIANG Z, YANG G, et al. A Robust Forgery Detection Algorithm for Object Removal by Exemplar-Based Image Inpainting[J]. Multimedia Tools and Applications, 2018, 77(10):11823-11842. doi: 10.1007/s11042-017-4829-0
[6]	NEWSON A, ALMANSA A, GOUSSEAU Y, et al. Non-Local Patch-Based Image Inpainting[J]. Image Processing on Line, 2017, 7:373-385. doi: 10.5201/ipol.2017.189
[7]	ZHANG N, JI H, LIU L, et al. Exemplar-Based Image Inpainting Using Angle-Aware Patch Matching[J]. EURASIP Journal on Image and Video Processing, 2019, 2019(1):70. doi: 10.1186/s13640-019-0471-2
[8]	YANG S, LIANG H, WANG Y, et al. Image Inpainting Based on Multi-Patch Match with Adaptive Size[J]. Applied Sciences, 2020, 10(14):4921. doi: 10.3390/app10144921
[9]	FAN Q, ZHANG L. A Novel Patch Matching Slgorithm for Exemplar-Based Image Inpainting[J]. Multimedia Tools and Applications, 2018, 77(9):10807-10821. doi: 10.1007/s11042-017-5077-z
[10]	ELHARROUSS O, ALMAADEED N, AL-MAADEED S, et al. Image Inpainting:A Review[J]. Neural Processing Letters, 2020, 51:2007-2028. doi: 10.1007/s11063-019-10163-0
[11]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Nets[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. New York: ACM, 2014:2672-2680.
[12]	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context Encoders:Feature Learning by Inpainting[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:2536-2544.
[13]	LIU G, REDA F A, SHIH K J, et al. Image Inpainting for Irregular Holes Using Partial Convolutions[C]// Proceedings of the European Conference on Computer Vision (ECCV).Heidelberg:Springer, 2018:89-105.
[14]	SAGONG M, SHIN Y, KIM S, et al. PEPSI:Fast Image Inpainting with Parallel Decoding Network[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2019:11360-11368.
[15]	NAZERI K, NG E, JOSEPH T, et al. EdgeConnect:Generative Image Inpainting with Adversarial Edge Learning (2019)[J/OL]. [2019-01-05]. https://arxiv.org/abs/1901.00212v2.
[16]	LIN X, MA L, LIU W, et al. Context-Gated Convolution (2019)[J/OL]. [2019-10-12]. https://arxiv.org/abs/1910.05577.
[17]	曹卫东, 李嘉琪, 王怀超. 采用注意力门控卷积网络模型的目标情感分析[J]. 西安电子科技大学学报, 2019, 46(6):30-36.
	CAO Weidong, LI Jiaqi, WANG Huaichao. Analysis of Targeted Sentiment by the Attention Gated Convolutional Network Model[J]. Journal of Xidian University, 2019, 46 (6):30-36.
[18]	YUSHCHENKO V, ARASLANOV N, ROTH S. Markov Decision Process for Video Generation[C]// Proceedings of the IEEE International Conference on Computer Vision Workshop.Piscataway:IEEE, 2019:1523-1532.
[19]	MEHTA S, RASTEGARI M, CASPI A, et al. ESPNet:Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation[C]// Proceedings of the European Conference on Computer Vision (ECCV).Heidelberg:Springer, 2018:561-580.
[20]	LOU L, ZANG S. Research on Edge Detection Method Based on Improved HED Network[J]. Journal of Physics:Conference Series, 2020, 1607:012068. doi: 10.1088/1742-6596/1607/1/012068
[21]	SRINIVASAN S, BABAKI B, FARNADI G, et al. Lifted Hinge-Loss Markov Random Fields[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2019, 33:7975-7983.
[22]	KIM J, KIM M, KANG H, et al. U-GAT-IT:Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation (2019)[J/OL]. [2019-07-25]. https://arxiv.org/abs/1907.10830v3.

掩码区域比例	CE (PSNR/SSIM)	Pconv (PSNR/SSIM)	PEPSI (PSNR/SSIM)	EdgeConnect (PSNR/SSIM)	文中方法 (PSNR/SSIM)
1%~10%	29.39/0.948	30.56/0.947	31.83/0.902	32.56/0.976	34.87/0.981
10%~20%	20.48/0.720	24.36/0.893	28.02/0.869	27.95/0.920	29.73/0.938
20%~30%	18.06/0.623	21.19/0.815	24.90/0.777	24.92/0.861	26.02/0.891
30%~40%	17.26/0.588	19.13/0.739	22.45/0.685	22.84/0.799	23.40/0.822
40%~50%	15.00/0.499	17.75/0.662	20.86/0.589	21.16/0.731	21.63/0.757
50%~60%	13.59/0.404	16.33/0.598	17.49/0.496	18.84/0.641	19.18/0.642
60%~70%	11.74/0.373	15.34/0.511	15.75/0.502	18.68/0.637	18.79/0.638
70%~80%	11.29/0.305	14.42/0.412	13.29/0.396	16.21/0.450	16.45/0.459
80%~90%	10.02/0.217	11.86/0.310	11.40/0.289	13.72/0.368	13.96/0.372

算法	修复速度/(帧·s^-1)	参数量
CE	4.6	68.2×10⁶
Pconv	3.1	179.7×10⁶
PEPSI	6.5	215.4×10⁶
EdgeConnect	6.8	103.6×10⁶
文中方法	7.7	105.1×10⁶