结合主题感知与通信代理的文本摘要模型

doi:10.19665/j.issn1001-2400.2020.03.014

摘要/Abstract

摘要：

针对传统自动文本摘要模型受循环神经网络长度限制而无法生成高质量的长文本摘要这一问题,提出了一种结合主题感知与通信代理的文本摘要模型。首先,将编码器划分为相互之间存在通信的多个代理,以解决长短期记忆网络输入序列较长而不能联合先验信息生成摘要的问题;然后,使用联合注意力机制加入主题信息,提高生成摘要与源文本的相关性;最后,使用带有强化学习的混合训练方法对模型进行训练,解决曝光偏差问题,直接对评价指标进行优化。实验结果表明,该模型不仅生成了主题突出的长文本摘要,并且得分比目前最先进的模型有一定提升。说明在主题信息的帮助下,该通信代理模型能够更好地生成长文本摘要。

关键词: 自动文本摘要, 通信代理, 主题感知, 联合注意力机制, 强化学习

Abstract:

To solve the problem that the traditional automatic text summary model cannot generate a high-quality long text summary due to the limitation of the length of the RNN (Recurrent Neural Network), a model of abstractive text summarization for topic-aware communicating agents has been proposed. First, the problem that the LSTM (Long Short-Term Memory) input sequence is too long to generate the abstract with prior information has been solved by dividing the encoder into multiple collaborating agents. Then for providing topic information and improving the correlation between the generated abstract and the source text, the joint attention mechanism has been added into our model. Finally, a hybrid training method with reinforcement learning has been employed in order to solve the problem of exposure bias, and optimize the evaluation index directly. The results show that our model not only generate long text summaries with prominent themes, but also has a higher score than the state-of-the-art models, which indicates that with the help of topic information, the model for communicating agents can be expected to generate long text summaries better.

Key words: abstractive text summarization, communicating agents, topic awareness, joint attention, reinforcement learning

中图分类号:

TP183

张哲铭,任淑霞,郭凯杰. 结合主题感知与通信代理的文本摘要模型[J]. 西安电子科技大学学报, 2020, 47(3): 97-104.

ZHANG Zheming,REN Shuxia,GUO Kaijie. Model of abstractive text summarization for topic-aware communicating agents[J]. Journal of Xidian University, 2020, 47(3): 97-104.

图/表 3

图1

表1

表2

参考文献 15

[1]	NALLAPATI R, ZHAI F, ZHOU B. Summarunner: a Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents[C]//Proceedings of the 2017 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2017: 3075-3081.
[2]	CELIKYILMAZ A, BOSSELUT A, HE X D, et al. Deep Communicating Agents for Abstractive Summarization[C/OL]. [2019-10-22].https://arxiv.org/pdf/1803.10357.pdf.
[3]	石磊, 阮选敏, 魏瑞斌, 等. 基于序列到序列模型的生成式文本摘要研究综述[J]. 情报学报, 2019,38(10):1102-1116.
	SHI Lei, RUAN Xuanmin, WEI Ruibin, et al. Abstractive Summarization Based on Sequence to Sequence Models: A Review[J]. Journal of the China Society for Scientific and Technical Information, 2019,38(10):1102-1116.
[4]	RUSH A M, CHOPRA S, WESTON J. A Neural Attention Model for Abstractive Sentence Summarization[C/OL]. [2019-10-22].https://arxiv.org/pdf/1509.00685.pdf.
[5]	SEE A, LIU P J, MANNING C D. Get to the Point: Summarization with Pointer-generator Networks[C/OL]. [2019-10-22].https://arxiv.org/pdf/1704.04368.pdf.
[6]	GEHRING J, AULI M, GRANGIER D, et al. Convolutional Sequence to Sequence Learning[C/OL]. [2019-10-22].https://arxiv.org/pdf/1705.03122.pdf.
[7]	李勇, 成红红, 梁新彦, 等. CNN图像标题生成[J]. 西安电子科技大学学报, 2019,46(2):152-157.
	LI Yong, CHENG Honghong, LIANG Xinyan, et al. CNN Image Caption Generation[J]. Journal of Xidian University, 2019,46(2):152-157.
[8]	XING C, WU W, WU Y, et al. Topic Aware Neural Response Generation[C]//Proceedings of the 2017 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2017: 3351-3357.
[9]	马书磊, 张国宾, 焦阳, 等. 一种改进的全局注意机制图像描述方法[J]. 西安电子科技大学学报, 2019,46(2):17-22.
	MA Shulei, ZHANG Guobin, JIAO Yang, et al. Improved Method for Image Caption with Global Attention Mechanism[J]. Journal of Xidian University, 2019,46(2):17-22.
[10]	PAULUS R, XIONG C, SOCHER R. A Deep Reinforced Model for Abstractive Summarization[C/OL]. [2019-10-22].https://arxiv.org/pdf/1705.04304.pdf.
[11]	RENNIE S J, MARCHERET E, MROUEH Y, et al. Self-critical Sequence TRAINING for Image Captioning [C]//Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 1179-1195.
[12]	HERMANN K M, KOCISKY T, GREFENSTETTE E, et al. Teaching Machines to Read and Comprehend [C]//Advances in Neural Information Processing Systems. Vancouver, Canada: Neural Information Processing Systems Foundation, 2015: 1693-1701.
[13]	LIN C Y, HOVY E. Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics [C]//Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: 1. New York: Association for Computational Linguistics, 2003: 71-78.
[14]	ZHOU Q, YANG N, WEI F, et al. Neural Document Summarization by Jointly Learning to Score and Select Sentences[C/OL]. [2019-10-22].https://arxiv.org/pdf/1807.02305.pdf.
[15]	CHEN Y C, BANSAL M. Fast Abstractive Summarization with Reinforce-selected Sentence Rewriting [C]//Proceedings of the 2018 56th Annual Meeting of the Association for Computational Linguistics. New York: Association for Computational Linguistics, 2018: 675-686.

基线模型	模型描述
SummaRuNNer^[1]	一个简单的基于RNN的抽取式模型,使用的是端到端的训练机制。此模型在CNN/DailyMail数据集上的ROUGE评分是2017年最高的
pointer-generator与pointer-generator +coverage^[5]	一个新颖的指针生成网络模型,解决了未登录词(OOV)问题。pointer-generator+coverage在此基础上又加入了覆盖机制,解决了摘要生成时可能出现的自我重复问题,提高了生成摘要的质量
ML+RL, with intra-attention^[10]	一个基于RNN同时结合了内注意力机制的模型。这篇文章第一次将强化学习的训练方法用于自动文本摘要模型的训练之中。ROUGE评分较之前的模型有明显的提升
PGN^[14]	一个端到端的抽取式文本摘要模型,将选择策略集成到打分模型中,解决了句子打分和句子选择这两部分割裂的问题
rnn-ext+abs+RL+rerank^[15]	一个准确快速的摘要模型,提出了一种用于生成式摘要的句子级别的强化学习技术,在不需要标注关键句子语料的情况下,实现抽取式和生成式的结合
DCA MLE+SEM+RL, mpgen, with-comm, with caa (3-agents)^[2]	一个结合了深度通信代理,指针网络以及语义联合的模型,并加入了强化学习的混合训练方式

模型	ROUGE-1	ROUGE-2	ROUGE-L
SummaRuNNer^[1] pointer-generator^[5] pointer-generator + coverage^[5] ML, with intra-attention^[10] RL, with intra-attention^[10] ML + RL, with intra-attention^[10] PGN^[14] rnn-ext + abs + RL + rerank^[15]	39.60 36.44 39.53 38.30 41.16 39.87 39.53 39.66	16.20 15.66 17.28 14.81 15.75 15.82 17.28 15.85	35.30 33.42 36.38 35.49 39.08 36.90 36.38 37.34
DCA MLE + SEM + RL, mpgen, with-comm, with caa (3-agents)^[2]	41.69	19.47	37.92
TADCA + ML (our baseline-1) TADCA + RL (our baseline-2) TADCA + MIX (our model)	41.11 43.88 42.97	18.15 20.03 20.53	37.05 40.57 38.72