一种改进的全局注意机制图像描述方法

doi:10.19665/j.issn1001-2400.2019.02.004

Abstract

Abstract:

Aiming at the lack of global information in existing attention based image caption methods, we propose an improved image caption method with global attention mechanism. The proposed method mimics the entire human perception process via designing a global feature extraction network to enhance the global context based on visual attention mechanism. This paper compares the proposed method with the existing attention based image caption technique under the same dataset and hyper parameters, and analyzes the influence of introducing the global feature. The results show that our method outperforms the existing technique in objective evaluations with the challenging Chinese caption dataset. In the subjective evaluation, in the meanwhile, the captions generated by the proposed method describes the image more accurately, vividly and diversely, and they are more close to the natural language.

Key words: image caption, attention mechanism, global feature, convolutional neural network, recurrent neural network

CLC Number:

TP37

MA Shulei,ZHANG Guobin,JIAO Yang,SHI Guangming. Improved method for image caption with global attention mechanism[J].Journal of Xidian University, 2019, 46(2): 17-22.

Figures/Tables 6

References 14

[1]	许强, 李伟, 占荣辉 , 等. 一种改进的卷积神经网络SAR目标识别算法[J]. 西安电子科技大学学报, 2018,45(5):177-183.
	XU Qiang, LI Wei, ZHAN Ronghui , et al. Improved Algorithm for SAR Target Recognition Based on the Convolutional Neural Network[J]. Journal of Xidian University, 2018,45(5):177-183.
[2]	VINVALS O, TOSHEV A, BENGIO S , et al. Show and Tell: a Neural Image Caption Generator[C]// Proceedings of the 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2015: 3156-3164.
[3]	SZEGEDY C, IOFFE S, VANHOUCKE V , et al. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning[C]//Proceedings of the 2017 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2017: 4278-4284.
[4]	YU D F, FU J L, MEI T , et al. Multi-Level Attention Networks for Visual Question Answering[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4187-4195.
[5]	FU J L, ZHENG H L, MEI T . Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-grained Image Recognition[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4476-4484.
[6]	DU W B, WANG Y L, QIAO Y . Recurrent Spatial-temporal Attention Network for Action Recognition in Videos[J]. IEEE Transactions on Image Processing, 2018,27(3):1347-1360. doi: 10.1109/TIP.2017.2778563
[7]	李敏宁, 张巧, 赵劼 . 视觉注意机制下的三维重建技术的改进[J]. 西安电子科技大学学报, 2014,41(5):192-196. doi: 10.3969/j.issn.1001-2400.2014.05.032
	LI Mimning, ZHANG Qiao, ZHAO Jie . Improvement on Three-dimensional Reconstruction of the Attention Mechanism[J]. Journal of Xidian University, 2014,41(5):192-196. doi: 10.3969/j.issn.1001-2400.2014.05.032
[8]	LU J S, XIONG C M, PARIKH D , et al. Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 3242-3250.
[9]	ANDERSON P, HE X D, BUEHLER C , et al. Bottom-up and Top-down Attention for Image Captioning and Visual Question Answering[J/OL]. [2018-06-20]. https://arxiv.org/abs/1707. 07998.
[10]	NAJEMNIK J, GEISLER W S . Optimal Eye Movement Strategies in Visual Search[J]. Nature, 2005,434(7031):387-391. doi: 10.1016/j.ajo.2005.04.009 pmid: 15772663
[11]	MIKOLOV T, SUTSKEVER I, CHEN K , et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Advances in Neural Information Processing Systems: 26. Vancouver, Canada: Neural Information Processing Systems Foundation, 2013: 104690.
[12]	PAPINENI K, ROUKOS S, WARD T , et al. BLEU: a Method for Automatic Evaluation of Machine Translation[C]// Proceedings of the 2002 40the Annual Meeting of the Association for Computational Linguistics. New York : ACL, 2002: 311-318.
[13]	DENKOWSKI M, LAVIE A . Meteor Universal: Language Specific Translation Evaluation for Any Target Language[C]// Proceedings of the 2014 Ninth Workshop on Statistical Machine Translation. New York : ACL, 2014: 376-380.
[14]	VEDANTAM R, ZITNICK C L, PARIKH D. CID Er: Consensus-based Image Description Evaluation[C]// Proceedings of the 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2015: 4566-4575.

	BLEU₁	BLEU₂	BLEU₃	BLEU₄	METEOR	ROUGEL	CIDEr
基于自底向上方法	0.780	0.662	0.561	0.476	0.379	0.637	1.409
全局注意机制方法	0.782	0.665	0.564	0.477	0.380	0.638	1.433

[1]	YU Haoyang,YIN Liang,LI Shufang,LV Shun. Recognition algorithm for the little sample radar modulation signal based on the generative adversarial network [J]. Journal of Xidian University, 2021, 48(6): 96-104.
[2]	DONG Ruchan,JIAO Licheng,ZHAO Jin,SHEN Weiyan. Application of the deep fusion mechanism in object detection of remote sensing images [J]. Journal of Xidian University, 2021, 48(5): 128-138.
[3]	SUN Yanjing,WEI Li,ZHANG Nianlong,YUN Xiao,DONG Kaiwen,GE Min,CHENG Xiaozhou,HOU Xiaofeng. Person re-identification method combining the DD-GAN and Global feature in a coal mine [J]. Journal of Xidian University, 2021, 48(5): 201-211.
[4]	ZHOU Peng,YANG Jun. Semantic segmentation of remote sensing images based on neural architecture search [J]. Journal of Xidian University, 2021, 48(5): 47-57.
[5]	YANG Yunhang,MIN Lianquan. Multi-scalefusion sketch recognition model by dilated convolution [J]. Journal of Xidian University, 2021, 48(5): 92-99.
[6]	CHEN Changchuan,WANG Haining,HUANG Lian,HUANG Tao,LI Lianjie,HUANG Xiangkang,DAI Shaosheng. Facial expression recognition based on local representation [J]. Journal of Xidian University, 2021, 48(5): 100-109.
[7]	SONG Jianfeng,MIAO Qiguang,WANG Chongxiao,XU Hao,YANG Jin. Multi-scale single object tracking based on the attention mechanism [J]. Journal of Xidian University, 2021, 48(5): 110-116.
[8]	ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network [J]. Journal of Xidian University, 2021, 48(5): 15-22.
[9]	HUI Haisheng,ZHANG Xueying,WU Zelin,LI Fenglian. Method for stroke lesion segmentation using the primary-auxiliary path attention compensation network [J]. Journal of Xidian University, 2021, 48(4): 200-208.
[10]	WANG Ping,JIANG Yuze,ZHAO Guanghui. Object detection based on the multiscale location Enhancement network [J]. Journal of Xidian University, 2021, 48(3): 85-90.
[11]	XU Bin,ZHANG Yongshun,ZHANG Qin,WANG Fuping,ZHENG Guimei. Radar HRRP target recognition based on the multiplicative RNN model [J]. Journal of Xidian University, 2021, 48(2): 49-54.
[12]	GUO Zekun,TIAN Long,HAN Ning,WANG Penghui,LIU Hongwei,CHEN Bo. Radar HRRP based few-shot target recognition with CNN-SSD [J]. Journal of Xidian University, 2021, 48(2): 7-14.
[13]	ZHANG Lu,SUN Rong,LIU Jingwei. Cloned piggybacking framework for distributed storage [J]. Journal of Xidian University, 2020, 47(6): 139-147.
[14]	CHENG Lei,WANG Yue,TIAN Chunna. Residual attention mechanism for visual tracking [J]. Journal of Xidian University, 2020, 47(6): 148-157.
[15]	HU Jianwei,ZHAO Wei,CUI Yanpeng,CUI Junjie. PHP code vulnerability mining technology based on theimproved ASTNN [J]. Journal of Xidian University, 2020, 47(6): 164-173.

Improved method for image caption with global attention mechanism

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 6

References 14

Related Articles 15

Metrics

Comments

Recommended 10