基于多模态融合的视觉问答传输注意网络

doi:10.16180/j.cnki.issn1007-7820.2022.12.010

Abstract

Abstract:

In view of the shortcomings of traditional visual question answering tasks that cannot fully capture the complex correlation between multi-modal features, this study proposes a visual question-and-answer transmission attention network based on multi-modal fusion. In the feature extraction part, GloVe word embedding + LSTM is used to extract problem features, and ResNet-152 network is adopted to extract image features. Multi-modal fusion is performed through a 3-layer transfer attention network to learn global multi-modal embedding information, which is then used to recalibrate the input features. In addition, a multi-modal transmission attention learning architecture is designed. Through overlapping calculations on the transmission network, the combined features focus on the fine-grained parts of the image and the question, which improves the accuracy of the predicted answer. The experimental results on the VQA v1.0 data set show that the overall accuracy of the model reaches 69.92%, which is improved to varying degrees compared with the accuracy of the other 5 mainstream visual question answering models, indicating the effectiveness of the model and robustness.

Key words: visual question answering, multi-modal features, combined features, multi-modal embedding, attention, transmission network, fine-grained, multi-modal fusion

CLC Number:

TP391

WANG Mao,PENG Yaxiong,LU Anjiang. Visual Question Answer Transmission Attention Network Based on Multi-Modal Fusion[J].Electronic Science and Technology, 2022, 35(12): 72-77.

Figures/Tables 7

Figure 1.

Figure 2.

Figure 3.

Table 1.

Table 2.

Table 3.

Figure 4.

References 19

[1]	Peng L, Yang Y, Zhang X, et al. Answer again: Improving VQA with cascaded-answering model[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 9(9):1-12. doi: 10.1109/TKDE.1997.649323
[2]	Chen C, Han D, Wang J. Multimodal encoder-decoder attention networks for visual question answering[J]. IEEE Access, 2020(8):35662-35671.
[3]	Yu Z, Yu J, Xiang C, et al. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(12):5947-5959. doi: 10.1109/TNNLS.2018.2817340 pmid: 29993847
[4]	Zeng D J, Zhou G H, Wang J. Residual self-attention for visual question answering[C]. Kuala Lumpur: Proceedings of the First International Conference on Electrical, Control and Instrumentation Engineering, 2019.
[5]	Li R Y, Jia J Y. Visual question answering with question representation update(QRU)[C]. Barcelona: Proceedings of the Thirtieth International Conference on Neural Information Processing Systems, 2016.
[6]	Lu J, Yang J, Batra D, et al. Hierarchical question-image co-attention for visual question answering[C]. Barcelona: Proceedings of the Thirtieth International Conference on Neural Information Processing Systems, 2016.
[7]	郑萌. 基于改进注意力机制模型的智能英语翻译方法研究[J]. 电子科技, 2020, 33(11):84-87.
	Zheng Meng. Research on intelligent English translation method based on improved attention mechanism model[J]. Electronic Science and Technology, 2020, 33(11):84-87.
[8]	和文杰, 刘敬彪, 潘勉, 等. 基于Attention-CTC的自然场景文本识别算法[J]. 电子科技, 2019, 32(12):32-36.
	He Wenjie, Liu Jingbiao, Pan Mian, et al. Natural scene text recognition algorithm based on Attention-CTC[J]. Electronic Science and Technology, 2019, 32(12):32-36.
[9]	Tian W D, He B, Wang N X, et al. Multi-channel co-attention network for visual question answering[C]. Glasgow: Proceedings of the International Joint Conference on Neural Networks, 2020.
[10]	Lao M R, Guo Y M, Wang H, et al. Cross-modal multistep fusion network with co-attention for visual question answering[J]. IEEE Access, 2018(6):31516-31524.
[11]	Yu Z, Yu J, Cui Y H, et al. Deep modular co-attention networks for visual question answering[C]. Long Beach: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[12]	Gao P, Jiang Z, You H, et al. Dynamic fusion with intra-and inter-modality attention flow for visual question answering[C]. Long Beach: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[13]	Pennington J, Socher R, Manning C D. Glove: Global vectors for word representation[C]. Doha: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014.
[14]	Gers F. Long short-term memory in recurrent neural networks[J]. Verlag Nicht Ermittelbar, 2001, 9(8):1735-1780.
[15]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]. Las Vegas: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[16]	Antol S, Agrawal A, Lu J, et al. Vqa: Visual question answering[C]. Santiago: Proceedings of the IEEE International Conference on Computer Vision, 2015.
[17]	Yang Z C, He X D, Gao J F, et al. Stacked attention networks for image question answering[C]. Las Vegas: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[18]	Xiong C, Merity S, Socher R. Dynamic memory networks for visual and textual question answering[C]. New York: Proceedings of the International Conference on Machine Learning, 2016.
[19]	Yu D F, Fu J L, Mei T, et al. Multi-level attention networks for visual question answering[C]. Honolulu: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

模型	Test-dev准确率/%				Test-std准确率/%
模型	Y/N	Num	Other	Overall	Overall
SAN^[17]	79.30	36.60	46.10	58.70	58.90
DMN+^[18]	80.50	36.80	48.30	60.30	60.40
QRU^[5]	82.30	37.00	47.70	60.70	60.80
HieCoAtt^[6]	79.70	38.70	51.70	61.80	62.06
MLAN^[19]	83.80	40.20	53.70	64.60	64.80
本文	87.04	42.47	59.77	69.79	69.92

模型		正确率/%
模型1		67.66
模型2		66.97
模型3		67.68
模型4	P=1	67.65
	P=2	67.69
	P=3	67.72
	P=4	67.65
本文		67.72

层数	正确率/%
n=1	67.58
n=2	67.70
n=3	67.72
本文	67.72

Visual Question Answer Transmission Attention Network Based on Multi-Modal Fusion

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 7

References 19

Related Articles 15

Metrics

Comments

Recommended 0

[1]	SHEN Ningjing,YUAN Jian. Crowd Counting Algorithm Based on Residual Dense Connection and Attention Fusion [J]. Electronic Science and Technology, 2022, 35(6): 6-12.
[2]	CHEN Jinhong,CHEN Wei,YIN Zhong. Semantic Segmentation of Streetscape Based on Improved ExfuseNet [J]. Electronic Science and Technology, 2022, 35(6): 28-34.
[3]	Chaowei LIN,Feifei LI,Qiu CHEN. Globaland Local Scene Representation Method Based on Deep Convolutional Features [J]. Electronic Science and Technology, 2022, 35(4): 20-27.
[4]	Lingyu JI,Yongbin GAO,Chenglu ZHAO,Xianhua TANG,Kaicheng XU,Jiacheng XU. CTA Segmentation Algorithm of Abdominal Artery Based on 3D Fully Convolutional Network [J]. Electronic Science and Technology, 2022, 35(3): 38-44.
[5]	LI Hui,WANG Yicheng. CNNCIFG-Attention Model for Text Sentiment Classifcation [J]. Electronic Science and Technology, 2022, 35(2): 46-51.
[6]	WANG Xialin,KAN Xiu,FAN Yixuan. Classification and Recognition of P300 Event-Related Potential Based on LSTM-Attention Network [J]. Electronic Science and Technology, 2022, 35(12): 10-16.
[7]	YU Runrun,JIANG Xiaoyan,ZHU Kaiying,JIANG Guanghao. Real-Time Image Semantic Segmentation Based on Contextual Attention Mechanism [J]. Electronic Science and Technology, 2022, 35(12): 57-63.
[8]	ZHANG Wei,LIU Na,JIANG Yang,LI Qingdu. Garbage Detection and Classification Based on YOLO Neural Network [J]. Electronic Science and Technology, 2022, 35(10): 45-50.
[9]	ZHANG Shisen,SUN Xiankun,YIN Ling,LI Shixi. Design of Text Title Generation Prototype System Based on Neural Network [J]. Electronic Science and Technology, 2021, 34(5): 35-41.
[10]	XUE Yongjie,JU Zhiyong. Fish Recognition Algorithm Based on Improved AlexNet [J]. Electronic Science and Technology, 2021, 34(4): 12-17.
[11]	DONG Bo,ZHOU Yan,WANG Yongxiong. Saliency Detection by Progressive Structural Receptive Field and Global Attention [J]. Electronic Science and Technology, 2021, 34(1): 23-30.
[12]	LI Fupeng,FU Dongxiang. Sentiment Analysis Method of Financial Text Based on Transformer Encoder [J]. Electronic Science and Technology, 2020, 33(9): 10-15.
[13]	ZHENG Meng. Research on Intelligent English Translation Method Based on Improved Attention Mechanism Model [J]. Electronic Science and Technology, 2020, 33(11): 84-87.
[14]	YANG Siqi,CHEN Wei,ZHANG Qingqing,XIE Ye. Fault Diagnosis Method of Transmission Network Based on Rule Reasoning [J]. Electronic Science and Technology, 2019, 32(3): 42-46.
[15]	HE Wenjie,LIU Jingbiao,PAN Mian,LÜ Shuaishuai. Natural Scene Text Recognition Algorithm Based on Attention-CTC [J]. Electronic Science and Technology, 2019, 32(12): 32-36.