一种改进的全局注意机制图像描述方法

doi:10.19665/j.issn1001-2400.2019.02.004

摘要/Abstract

摘要：

针对现有基于注意机制的图像描述方法全局信息缺失问题,提出了一种改进的全局注意机制图像描述方法。该方法在注意机制的基础上,通过设计全局特征网络来模拟人类感知机制的全过程,对图像全局特征进行增强。将所提方法在相同数据集和网络超参数的情况下与目前最优网络进行实验对比,分析了全局信息对生成文本的影响。实验结果显示,文中提出的方法在更具挑战性的中文文本描述任务上客观评价指标优于目前最优的模型。同时,在主观评价中能够生成更准确的文本内容,也更具丰富性与多样性,接近自然语言描述。

关键词: 图像描述, 注意力机制, 全局特征, 卷积神经网络, 循环神经网络

Abstract:

Aiming at the lack of global information in existing attention based image caption methods, we propose an improved image caption method with global attention mechanism. The proposed method mimics the entire human perception process via designing a global feature extraction network to enhance the global context based on visual attention mechanism. This paper compares the proposed method with the existing attention based image caption technique under the same dataset and hyper parameters, and analyzes the influence of introducing the global feature. The results show that our method outperforms the existing technique in objective evaluations with the challenging Chinese caption dataset. In the subjective evaluation, in the meanwhile, the captions generated by the proposed method describes the image more accurately, vividly and diversely, and they are more close to the natural language.

Key words: image caption, attention mechanism, global feature, convolutional neural network, recurrent neural network

中图分类号:

TP37

马书磊,张国宾,焦阳,石光明. 一种改进的全局注意机制图像描述方法[J]. 西安电子科技大学学报, 2019, 46(2): 17-22.

MA Shulei,ZHANG Guobin,JIAO Yang,SHI Guangming. Improved method for image caption with global attention mechanism[J]. Journal of Xidian University, 2019, 46(2): 17-22.

图/表 6

图1

图2

图3

图4

表1

图5

参考文献 14

[1]	许强, 李伟, 占荣辉 , 等. 一种改进的卷积神经网络SAR目标识别算法[J]. 西安电子科技大学学报, 2018,45(5):177-183.
	XU Qiang, LI Wei, ZHAN Ronghui , et al. Improved Algorithm for SAR Target Recognition Based on the Convolutional Neural Network[J]. Journal of Xidian University, 2018,45(5):177-183.
[2]	VINVALS O, TOSHEV A, BENGIO S , et al. Show and Tell: a Neural Image Caption Generator[C]// Proceedings of the 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2015: 3156-3164.
[3]	SZEGEDY C, IOFFE S, VANHOUCKE V , et al. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning[C]//Proceedings of the 2017 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2017: 4278-4284.
[4]	YU D F, FU J L, MEI T , et al. Multi-Level Attention Networks for Visual Question Answering[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4187-4195.
[5]	FU J L, ZHENG H L, MEI T . Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-grained Image Recognition[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4476-4484.
[6]	DU W B, WANG Y L, QIAO Y . Recurrent Spatial-temporal Attention Network for Action Recognition in Videos[J]. IEEE Transactions on Image Processing, 2018,27(3):1347-1360. doi: 10.1109/TIP.2017.2778563
[7]	李敏宁, 张巧, 赵劼 . 视觉注意机制下的三维重建技术的改进[J]. 西安电子科技大学学报, 2014,41(5):192-196. doi: 10.3969/j.issn.1001-2400.2014.05.032
	LI Mimning, ZHANG Qiao, ZHAO Jie . Improvement on Three-dimensional Reconstruction of the Attention Mechanism[J]. Journal of Xidian University, 2014,41(5):192-196. doi: 10.3969/j.issn.1001-2400.2014.05.032
[8]	LU J S, XIONG C M, PARIKH D , et al. Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning[C]// Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 3242-3250.
[9]	ANDERSON P, HE X D, BUEHLER C , et al. Bottom-up and Top-down Attention for Image Captioning and Visual Question Answering[J/OL]. [2018-06-20]. https://arxiv.org/abs/1707. 07998.
[10]	NAJEMNIK J, GEISLER W S . Optimal Eye Movement Strategies in Visual Search[J]. Nature, 2005,434(7031):387-391. doi: 10.1016/j.ajo.2005.04.009 pmid: 15772663
[11]	MIKOLOV T, SUTSKEVER I, CHEN K , et al. Distributed Representations of Words and Phrases and Their Compositionality[C]// Advances in Neural Information Processing Systems: 26. Vancouver, Canada: Neural Information Processing Systems Foundation, 2013: 104690.
[12]	PAPINENI K, ROUKOS S, WARD T , et al. BLEU: a Method for Automatic Evaluation of Machine Translation[C]// Proceedings of the 2002 40the Annual Meeting of the Association for Computational Linguistics. New York : ACL, 2002: 311-318.
[13]	DENKOWSKI M, LAVIE A . Meteor Universal: Language Specific Translation Evaluation for Any Target Language[C]// Proceedings of the 2014 Ninth Workshop on Statistical Machine Translation. New York : ACL, 2014: 376-380.
[14]	VEDANTAM R, ZITNICK C L, PARIKH D. CID Er: Consensus-based Image Description Evaluation[C]// Proceedings of the 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington: IEEE Computer Society, 2015: 4566-4575.

	BLEU₁	BLEU₂	BLEU₃	BLEU₄	METEOR	ROUGEL	CIDEr
基于自底向上方法	0.780	0.662	0.561	0.476	0.379	0.637	1.409
全局注意机制方法	0.782	0.665	0.564	0.477	0.380	0.638	1.433

[1]	李源,崔玉爽,王伟. 一种基于字词双通道网络的文本情感分析方法[J]. 西安电子科技大学学报, 2021, 48(6): 179-186.
[2]	于浩洋,尹良,李书芳,吕顺. 生成对抗网络小样本雷达调制信号识别算法[J]. 西安电子科技大学学报, 2021, 48(6): 96-104.
[3]	董如婵,焦李成,赵进,沈维燕. 一种深度融合机制的遥感图像目标检测技术[J]. 西安电子科技大学学报, 2021, 48(5): 128-138.
[4]	孙彦景,魏力,张年龙,云霄,董锴文,葛敏,程小舟,侯晓峰. 联合DD-GAN和全局特征的井下人员重识别方法[J]. 西安电子科技大学学报, 2021, 48(5): 201-211.
[5]	周鹏,杨军. 采用神经网络架构搜索的遥感影像分割方法[J]. 西安电子科技大学学报, 2021, 48(5): 47-57.
[6]	杨云航,闵连权. 采用空洞卷积的多尺度融合草图识别模型[J]. 西安电子科技大学学报, 2021, 48(5): 92-99.
[7]	陈昌川,王海宁,黄炼,黄涛,李连杰,黄向康,代少升. 一种基于局部表征的面部表情识别算法[J]. 西安电子科技大学学报, 2021, 48(5): 100-109.
[8]	宋建锋,苗启广,王崇晓,徐浩,杨瑾. 注意力机制的多尺度单目标跟踪算法[J]. 西安电子科技大学学报, 2021, 48(5): 110-116.
[9]	张宇浩,程培涛,张书豪,王秀美. 一种自适应权重学习的轻量超分辨率重建网络[J]. 西安电子科技大学学报, 2021, 48(5): 15-22.
[10]	回海生,张雪英,吴泽林,李凤莲. 一种主辅路径注意力补偿的脑卒中病灶分割方法[J]. 西安电子科技大学学报, 2021, 48(4): 200-208.
[11]	王平,江雨泽,赵光辉. 目标检测的多尺度定位提升算法[J]. 西安电子科技大学学报, 2021, 48(3): 85-90.
[12]	徐彬,张永顺,张秦,王富平,郑桂妹. 采用乘性RNN的雷达HRRP目标识别[J]. 西安电子科技大学学报, 2021, 48(2): 49-54.
[13]	郭泽坤,田隆,韩宁,王鹏辉,刘宏伟,陈渤. 采用CNN-SSD的雷达HRRP小样本目标识别方法[J]. 西安电子科技大学学报, 2021, 48(2): 7-14.
[14]	张璐,孙蓉,刘景伟. 分布式存储中的克隆piggybacking框架[J]. 西安电子科技大学学报, 2020, 47(6): 139-147.
[15]	成磊,王玥,田春娜. 一种添加残差注意力机制的视觉目标跟踪算法[J]. 西安电子科技大学学报, 2020, 47(6): 148-157.