关键词: 视觉问答, 多模态特征, 组合特征, 多模态嵌入, 注意力, 传输网络, 细粒度, 多模态融合

Abstract:

In view of the shortcomings of traditional visual question answering tasks that cannot fully capture the complex correlation between multi-modal features, this study proposes a visual question-and-answer transmission attention network based on multi-modal fusion. In the feature extraction part, GloVe word embedding + LSTM is used to extract problem features, and ResNet-152 network is adopted to extract image features. Multi-modal fusion is performed through a 3-layer transfer attention network to learn global multi-modal embedding information, which is then used to recalibrate the input features. In addition, a multi-modal transmission attention learning architecture is designed. Through overlapping calculations on the transmission network, the combined features focus on the fine-grained parts of the image and the question, which improves the accuracy of the predicted answer. The experimental results on the VQA v1.0 data set show that the overall accuracy of the model reaches 69.92%, which is improved to varying degrees compared with the accuracy of the other 5 mainstream visual question answering models, indicating the effectiveness of the model and robustness.

Key words: visual question answering, multi-modal features, combined features, multi-modal embedding, attention, transmission network, fine-grained, multi-modal fusion