基于深度学习与显著性的数字图像构图优化

doi:10.16180/j.cnki.issn1007-7820.2021.03.007

摘要/Abstract

摘要：

构图是决定数字图像美学质量的重要因素,而现有的计算机优化算法在这一领域还存在整体性、视觉平衡感不足等问题。针对这一问题,文中提出了一种基于深度学习模型的构图优化算法,将显著特征与改善视觉平衡相结合,实现了更符合图像美学的构图优化。文中卷积网络以VGG-16为主干,加权两项损失函数并以图像像素进行平均,可在训练后实现端到端的全分辨率显著性回归。定性对比和定量验证的实验结果表明,相较于传统优化方法,文中算法在改善视觉平衡方面具有明显优势,处理后的图像画面平衡感得到显著提升,更符合美学评价原则和人的视觉感受。

关键词: 图像美学, 构图优化, 深度学习, 显著性, 端到端模型, 视觉平衡

Abstract:

Image composition is an important factor in the digital image aesthetic quality. Existing computer optimization algorithms have a number of shortcomings, such as visual imbalance and poor integrity. Therefore, this paper proposes a novel composition optimization algorithm based on the deep learning model. This algorithm combines salient features with improved visual balance to achieve composition optimization that is more in line with image aesthetics. The deep convolutional neural network is based on VGG-16, weighting two loss functions and averaging image pixel values. After training, the network could achieve end-to-end saliency regression without any pre-processing or post-processing. Comparative experiments and quantitative verification results demonstrate that compared with the current traditional optimization approaches, the proposed algorithm has obvious advantages in improving visual balance. The processed image has a significantly improved sense of balance, which is more in line with the principles of aesthetic evaluation and visual perception.

Key words: image aesthetic, composition optimization, deep learning, salient feature, end-to-end model, visual balance

中图分类号:

TP391

邵杭,王永雄,秦宇龙. 基于深度学习与显著性的数字图像构图优化[J]. 电子科技, 2021, 34(3): 36-42.

SHAO Hang,WANG Yongxiong,QIN Yulong. Digital Image Composition Optimization Based on Salient Feature Algorithm[J]. Electronic Science and Technology, 2021, 34(3): 36-42.

图/表 12

图1

图2

图3

图4

图5

图6

表1

图7

表2

图8

表3

图9

参考文献 31

[1]	Neumann L, Sbert M, Gooch B. Defining computational aesthetics[J]. Computational Aesthetics in Graphics, Visualization and Imaging, 2005(5):13-18.
[2]	王伟凝, 蚁静缄, 贺前华. 可计算图像美学研究进展[J]. 中国图象图形学学报, 2012,17(8):893-901.
	Wang Weining, Yi Jingjian, He Qianhua. Review for computational image aesthetics[J]. Journal of Image and Graphics, 2012,17(8):893-901.
[3]	Deng Y, Loy C, Tang X. Image aesthetic assessment: An experimental survey[J]. IEEE Signal Processing Magazine, 2017,34(4):80-106.
[4]	胡少聪. 基于深度学习的人脸识别方法研究[J]. 电子科技, 2019,32(6):82-86.
	Hu Shaocong. Research on face recognition based on deep learning[J]. Electronic Science and Technology, 2019,32(6):82-86.
[5]	Lu P, Zhang H, Peng X. Aesthetic guided deep regression network for image cropping[J]. Signal Processing Image Communication, 2019,77(6):1-10.
[6]	Kong S, Shen X, Lin Z, et al. Photo aesthetics ranking network with attributes and content adaptation[C]. Amsterdam:Proceedings of European Conference on Computer Vision, 2016.
[7]	Wang W, Shen J. Deep cropping via attention box prediction and aesthetics assessment[C]. Venice: Proceedings of IEEE International Conference on Computer Vision, 2017.
[8]	Zhu J, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]. Venice:Proceedings of IEEE International Conference on Computer Vision, 2017.
[9]	Levada A. Closed-form Bayesian image denoising: Improving the adaptive Wiener filter through pairwise Gaussian-Markov random fields[J]. Communication in Statistics Simulation and Computation, 2019(7):1-25.
[10]	Ignatov A, Kobyshev N, Timofte R, et al. DSLR-quality photos on mobile devices with deep convolutional networks[C]. Venice:Proceedings of IEEE International Conference on Computer Vision, 2017.
[11]	Yao L, Suryanarayan P, Qiao M, et al. Oscar:On-site composition and aesthetics feedback through exemplars for photographers[J]. International Journal of Computer Vision, 2012,96(9):353-383. doi: 10.1007/s11263-011-0478-3
[12]	顾婷婷, 郭延文, 殷昆燕. 结合浅景深与构图的图像质量评价[J]. 中国图象图形学学报, 2013,18(5):574-582. doi: 10.11834/jig.20130512
	Gu Tingting, Guo Yanwen, Yin Kunyan. Image quality assessment combining low DoF and composition[J]. Journal of Image and Graphics, 2013,18(5):574-582. doi: 10.11834/jig.20130512
[13]	袁小平, 王岗, 王晔枫, 等. 基于改进卷积神经网络的交通标志识别方法[J]. 电子科技, 2019,32(11):28-32.
	Yuan Xiaoping, Wang Gang, Wang Yefeng, et al. Traffic sign recognition method based on improved convolutional neural network[J]. Electronic Science and Technology, 2019,32(11):28-32.
[14]	凌艳, 陈莹. 多尺度上下文信息增强的显著目标检测全卷积网络[J]. 计算机辅助设计与图形学学报, 2019,31(11):2007-2016.
	Ling Yan, Chen Ying. Salient object detection with multiscale context enhanced fully convolutional network[J]. Journal of Computer-Aided Design & Computer Graphics, 2019,31(11):2007-2016.
[15]	Bhattacharya S, Sukthankar R, Shah M. A framework for photo quality assessment and enhancement based on visual aesthetics[C]. Firenze:Proceedings of ACM International Conference on Multimedia, 2010.
[16]	Jin Y, Wu Q, Liu L. Aesthetic photo composition by optimal crop-and-warp[J]. Computers and Graphics, 2012,36(8):955-965. doi: 10.1016/j.cag.2012.07.007
[17]	Guo Y, Liu M, Gu T, et al. Improving photo composition elegantly:Considering image similarity during composition optimization[J]. Computer Graphics Forum, 2012,31(2):193-202.
[18]	Zhang F, Wang M. Aesthetic image enhancement by dependence-aware object recomposition[J]. IEEE Transactions on Multimedia, 2013,15(7):1480-1490. doi: 10.1109/TMM.2013.2268051
[19]	王伟凝, 刘剑聪, 徐向民, 等. 基于构图规则的图像美学优化[J]. 华南理工大学学报(自然科学版), 2015(5):51-58.
	Wang Weining, Liu Jiancong, Xu Xiangmin, et al. Aesthetic enhancement of images based on photography composition guidelines[J]. Journal of South China University of Technology (Natural Science Edition), 2015(5):51-58.
[20]	熊杨超. 图像美学评价及美学优化研究[D]. 广州:华南理工大学, 2015.
	Xiong Yangchao. Research on image aesthetic assessment and optimizing[D]. Guangzhou: South China University of Technology, 2015.
[21]	Chen J, Bai G, Liang S, et al. Automatic image cropping: A computational complexity study[C]. Las Vegas: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[22]	Chen Y, Huang T, Chang K, et al. Quantitative analysis of automatic image cropping algorithms: A dataset and comparative study[C]. Santa Rosa:Proceedings of IEEE Winter Conference on Applications of Computer Vision, 2017.
[23]	Wang W, Shen J, Ling H. A deep network solution for attention and aesthetics aware photo cropping[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019,41(9):1531-1544. doi: 10.1109/TPAMI.34
[24]	何援军. 透视和透视投影变换—论图形变换和投影的若干问题之三[J]. 计算机辅助设计与图形学学报, 2005,17(4):735-739.
	He Yuanjun. Perspective and its projection transformation[J]. Journal of Computer Aided Design and Computer Graphics, 2005,17(4):735-739.
[25]	Liu T, Sun J, Zheng N, et al. Learning to detect a salient object[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011,33(2):353-367. pmid: 21193811
[26]	Ke Y, Tang X, Jing F. The design of high-level features for photo quality assessment[C]. New York:Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2006.
[27]	Luo W, Wang X, Tang X. Content-based photo quality assessment[J]. IEEE Transactions on Multimedia, 2013(5):1930-1943.
[28]	Murray N, Marchesotti L, Perronnin F. AVA: A large scale database for aesthetic visual analysis[C]. Rhode Island:Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012.
[29]	Malu G, Bapi R, Indurkhya B. Learning photography aesthetics with deep CNNs[C]. Fort Wayne:Proceedings of Modern Artificial Intelligence and Cognitive Science, 2017.
[30]	Zhu W, Liang S, Wei Y, et al. Saliency optimization from robust background detection[C]. Columbus:Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[31]	Li H, Lu H, Lin Z, et al. Inner and inter label propagation: salient object detection in the wild[J]. IEEE Transactions on Image Processing, 2015,24(4):3176-3186. doi: 10.1109/TIP.83

方法	ℓ₁损失	ℓ₂损失	PSNR	SSIM
wCtr	14.06%	18.34%	10.97	79.93%
LPS	17.01%	18.49%	9.98	75.21%
本文	16.13%	18.68%	9.13	72.81%

方法	ℓ₁损失	ℓ₂损失	PSNR	SSIM
wCtr	8.26%	9.42%	11.62	82.12%
LPS	8.16%	9.74%	12.14	83.24%
本文	6.29%	7.87%	12.69	89.45%

方法	ℓ₁损失	ℓ₂损失	PSNR	SSIM
文献[17]	18.58%	10.21%	19.74	78.04%
文献[19]	22.95%	13.74%	15.65	59.24%
文献[22]	6.63%	8.69%	12.57	85.19%
本文	6.29%	7.87%	12.69	89.45%