短文本新闻标题生成方法

doi:10.16180/j.cnki.issn1007-7820.2024.09.013

摘要/Abstract

摘要：

当今新闻具有文本短、发布频繁、时效性强等特点,一个媒体账号一天内发布数十条新闻。为大量新闻制定适用且有吸引力的标题已经成为媒体工作者的一项主要工作内容。媒体工作者需要一个自动生成短文本标题的系统来缓解工作压力。为解决该问题,文中提出了一种短文本新闻标题生成模型。该模型采用序列到序列结构,在编码器和解码器分别应用预训练语言模型和分层自注意力解码器。为了使生成标题包含原始新闻的关键信息,提出一种基于LCSTS数据集和Weibo4数据集的分阶段训练方法,并使模型分别从这两个数据集学习提取关键新闻信息和构建风格化表达,使模型生成标题能够准确表达新闻的核心内容从而吸引读者。

关键词: 新闻标题生成, 预训练语言模型, 分层自注意力解码器, 编码器, 文本提取, 文本生成

Abstract:

Today's news has the characteristics of short text, frequent release, timeliness, etc. A media account releases dozens of news in a day. Developing suitable and attractive headlines for large volumes of news has become a major part of the work of media workers. Media workers need a system that automatically generates short text headlines to relieve their stress. To solve this problem, this study proposes a short text news title generation model. The model adopts sequence-to-sequence structure, using pre-trained language model and layered self-attention decoder in encoder and decoder respectively. In order to make the generated headlines contain the key information of the original news, a staged training method based on LCSTS data set and Weibo4 data set is proposed, and the model learns to extract the key news information and construct a stylized expression from the two data sets respectively, so that the generated headlines can accurately express the core content of the news and attract readers.

Key words: news headline generation, pre-training language model,, layered self-attention decoder, encoder, text extraction, text generation

中图分类号:

TP391

赵明. 短文本新闻标题生成方法[J]. 电子科技, 2024, 37(9): 87-94.

ZHAO Ming. Researchon Short Text News Title Generation Method[J]. Electronic Science and Technology, 2024, 37(9): 87-94.

图/表 8

图1

图2

图3

图4

表1

表2

表3

图5

参考文献 20

[1]	汪家伟, 余晓. 基于深度学习的文本分类研究综述[J]. 电子科技, 2024, 37(1):81-86.
	Wang Jiawei, Yu Xiao. Review of text classification research based on deep learning[J]. Electronic Science and Technology, 2024, 37(1):81-86.
[2]	Devlin J, Chang MW, Lee K. BERT:Pre-training of deep bidirectional transformers for language understanding[C]. Minneapolis:Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[3]	Vaswani A, Shazeer N, Parmar N. Attention is all you need[C]. Long Beach: Proceedings of the Thirty-first International Conference on Neural Information Processing Systems, 2017:6000-6010.
[4]	张琪, 范永胜. 基于改进T5 PEGASUS模型的新闻文本摘要生成[J]. 电子科技, 2023, 36(12):72-78.
	Zhang Qi, Fan Yongsheng. Research on generating news text summarrization based on improved T5 PEGASUS model[J]. Electronic Science and Technology, 2023, 36(12):72-78.
[5]	Mihalcea R, Tarau P. Textrank:Bringing order into text[C]. Doha: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2004:404-411.
[6]	Page L, Brin S, Motwani R. The PageRank citation ranking:Bringing order to the web[M]. Palo Alto: Stanford InfoLab, 1999:61-89.
[7]	何传鹏, 尹玲, 黄勃, 等. 基于BERT和LightGBM的文本关键词提取方法[J]. 电子科技, 2023, 36(3):7-13.
	He Chuanpeng, Yin Ling, Huang Bo, et al. Text keyword extraction method based on BERT and LightGBM[J]. Electronic Science and Technology, 2023, 36(3):7-13.
[8]	Cheng J, Lapata M. Neural summarization by extracting sentences and words[C]. Berlin:Proceedings of the Fifty-fourth Annual Meeting of the Association for Computational Linguistics, 2016:484-494.
[9]	Nallapati R, Zhai F, Zhou B. Summarunner:A recurrent neural network based sequence model for extractive summarization of documents[C]. San Francisco: Proceedings of the AAAI Conference on Artificial Intelligence, 2017:311-318.
[10]	Narayan S, Cohen S B, Lapata M. Ranking sentences for extractive summarization with reinforcement learning[C]. New Orleans:Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018:1747-1759.
[11]	Zhou L, Hovy E. Template-filtered headline summarization[C]. Barcelona: Proceedings of the Workshop on Text Summarization Branches Out, 2004:56-60.
[12]	Rush A M, Chopra S, Weston J. A neural attention model for abstractive sentence summarization[C]. Lisbon: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2015:379-389.
[13]	李志欣, 彭智, 唐素勤, 等. 融合上下文信息和关键信息的文本摘要[J]. 中文信息学报, 2022, 36(1): 83-91.
	Li Zhixin, Peng Zhi, Tang Suqin, et al. Fusing context information and key information for text summarization[J]. Journal of Chinese Information Processing, 2022, 36(1):83-91.
[14]	See A, Liu P J, Manning C D. Get to the point:Summarization with pointer-generator networks[C]. Vancouver: Proceedings of the Fifty-fifth Annual Meeting of the Association for Computational Linguistics, 2017:1073-1083.
[15]	Lin J, Sun X, Ma S. Global encoding for abstractive summarization[C]. Melbourne: Proceedings of the Fifty-sixth Annual Meeting of the Association for Computational Linguistics, 2018:163-169.
[16]	Mikolov T, Sutskever I, Chen K. Distributed representations of words and phrases and their compositionality[C]. Nevada: Proceedings of the Twenty-sixth International Conference on Neural Information Processing Systems, 2013:3111-3119.
[17]	Pennington J, Socher R, Manning C D. Glove:Global vectors for word representation[C]. Doha: Proceedings of the Conference on Empirical Mmethods in Natural Language Processing, 2014:1532-1543.
[18]	Peters M, Neumann M, Iyyer M. Deep contextualized word representations[C]. New Orleans:Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018:2227-2237.
[19]	Xu Yueshen. Collaborative APIs recommendation for artificial intelligence of things with information fusion[J]. Future Generation Computer Systems, 2021, 12(5):471-479.
[20]	Loshchilov I, Hutter F. Decoupled weight decay regularization[C]. Vancouver: International Conference on Learning Representations, 2018:452-459.

模型	R-1	R-2	R-L
RNN	21.50	8.90	18.60
RNN-Context	29.90	17.40	27.20
CopyNet	34.40	21.60	31.30
Seq2Seq+CGU	39.40	26.90	36.50
BERT+TransformerDecoder	40.48	25.37	38.08

模型	R-1	R-2	R-L
①BERT(all-freezed)+ TransformerDecoder	34.35	19.93	31.32
②BERT(two-layer-unfreezed)+ TransformerDecoder	35.18	20.48	32.90
③BERT(all-unfreezed)+ TransformerDecoder	40.48	25.37	38.08

	Rouge_LCSTS			Rouge_Weibo4			Rouge_AVG
	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
①Weibo4_only	28.68	13.47	25.91	44.92	31.17	42.77	36.80	22.32	34.34
②LCSTS_only	40.48	25.37	38.08	39.25	25.61	37.61	39.87	25.49	37.85
③Stagewise	40.02	25.24	37.93	41.84	27.64	39.99	40.93	26.44	38.96