基于时序行为检测的工作流识别

doi:10.16180/j.cnki.issn1007-7820.2021.08.003

电子科技 ›› 2021, Vol. 34 ›› Issue (8): 14-18.doi: 10.16180/j.cnki.issn1007-7820.2021.08.003

基于时序行为检测的工作流识别

王庆文,胡海洋

杭州电子科技大学计算机学院,浙江杭州 310018

收稿日期:2020-03-26 出版日期:2021-08-15 发布日期:2021-08-17
作者简介:王庆文(1994-),男,硕士研究生。研究方向:工作流识别、计算机视觉。|胡海洋(1977-),男,博士,教授。研究方向:工作流调度、云计算、移动计算和分布式计算。
基金资助:
国家自然科学基金(61572162);国家自然科学基金(61272188);国家自然科学基金(61702144);浙江省重点研发计划(2018C01012);浙江省自然科学基金(LQ17F020003)

Workflow Recognition Based on Temporal Action Detection

WANG Qingwen,HU Haiyang

Computer & Software School, Hangzhou Dianzi University,Hangzhou 310018,China

Received:2020-03-26 Online:2021-08-15 Published:2021-08-17
Supported by:
National Natural Science Foundation of China(61572162);National Natural Science Foundation of China(61272188);National Natural Science Foundation of China(61702144);Zhejiang Provincial Key Science and Technology Project Foundation(2018C01012);Natural Science Foundation of Zhejiang Province(LQ17F020003)

摘要/Abstract

摘要：

在智能制造环境中,基于动作识别的工作流识别方法难以定位出视频中工作流活动的开始和结束时间。为了从视频中对工作流中的活动进行时序定位,文中对R-C3D网络模型进行改进并提出了一种基于时序行为检测的工作流识别方法。在文中所提出的工作流识别方法中,采用一种随机稀疏采样策略来减少相邻视频帧的冗余,并使用Res3D网络来提取视频中的时空特征。此外,所提方法通过Soft-NMS消除了高度重叠以及低置信度的提名片段。实验结果证明,相比于R-C3D,文中方法在保证平均精度均值的同时,提高了约80%的训练速度和85%的检测速度,证明了该方法对于工作流识别的有效性。

关键词: 智能制造, 工作流识别, 动作识别, 时序行为检测, 时序定位, 稀疏采样, 时空特征, Soft-NMS

Abstract:

In a complex manufacturing environment, it is difficult for workflow recognition based on action recognition to localize the start and end times of each activity in a workflow from the video. In order to localize the activities temporally in the workflow from the video, according to R-C3D, a workflow recognition framework based on temporal action detection is proposed. The proposed workflow recognition method introduces a random sparse sampling strategy to reduce the redundancy between adjacent frames, and uses Res3D network to extract spatio-temporal features in the video. In addition, Soft-NMS strategy is used to eliminate the highly overlapping and low confidence proposals. Experiment results show that compared with R-C3D, the proposed method can improve the speed of training and detecting about 80% and 85%, respectively without loss of accuracy, proving the effectiveness of the method for workflow identification.

Key words: intelligent manufacturing, workflow recognition, action recognition, temporal action detection, temporal localization, sparse sampling, spatio-temporal features, Soft-NMS

中图分类号:

TP312

王庆文,胡海洋. 基于时序行为检测的工作流识别[J]. 电子科技, 2021, 34(8): 14-18.

WANG Qingwen,HU Haiyang. Workflow Recognition Based on Temporal Action Detection[J]. Electronic Science and Technology, 2021, 34(8): 14-18.

图/表 4

参考文献 17

[1]	Jin Y, Dou Q, Chen H, et al. SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network[J]. IEEE Transactions on Medical Imaging, 2017,37(5):1114-1126.
[2]	Yang H, Park M, Cho M, et al. A system architecture for manufacturing process analysis based on big data and process mining techniques[C]. Santa Clara:IEEE International Conference on Big Data, 2015.
[3]	袁小平, 王岗, 王晔枫, 等. 基于改进卷积神经网络的交通标志识别方法[J]. 电子科技, 2019,32(11):28-32.
	Yuan Xiaoping, Wang Gang, Wang Yefeng, et al. Traffic sign recognition method based on improved convolutional neural network[J]. Electronic Science and Technology, 2019,32(11):28-32.
[4]	周文凯, 韩芳, 孔维健. 基于Faster-RCNN的极验点选式验证码识别[J]. 电子科技, 2019,32(9):42-45,59.
	Zhou Wenkai, Han Fang, Kong Weijian. Point-selective geetest CAPTCHA recognition based on Faster-RCNN[J]. Electronic Science and Technology, 2019,32(9):42-45,59.
[5]	胡海洋, 丁佳民, 胡华, 等. 基于三维卷积神经网络的工作流识别方法[J]. 计算机集成制造系统, 2018,24(7):1747-1757.
	Hu Haiyang, Ding Jiamin, Hu Hua, et al. Workflow recognition method based on 3D convolutional neural networks[J]. Computer Integrated Manufacturing Systems, 2018,24(7):1747-1757.
[6]	Slama R, Wannous H, Daoudi M, et al. Accurate 3D action recognition using learning on the Grassmann manifold[J]. Pattern Recognition, 2015,48(2):556-567.
[7]	Xu H, Das A, Saenko K. R-C3D: Region convolutional 3d network for temporal activity detection[C]. Venice: The IEEE International Conference on Computer Vision, 2017.
[8]	Hu H, Cheng K, Li Z, et al. Workflow recognition with structured two-stream convolutional networks[J]. Pattern Recognition Letters, 2020,130(2):267-274.
[9]	Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos[C]. Montreal: Advances in Neural Information Processing Systems, 2014.
[10]	Bodla N, Singh B, Chellappa R, et al. Soft-NMS--improving object detection with one line of code[C]. Venice: The IEEE International Conference on Computer Vision, 2017.
[11]	Wang L, Xiong Y, Zhe W, et al. Temporal segment networks: towards good practices for deep action recognition[C]. Amsterdam:European Conference on Computer Vision, 2016.
[12]	Hara K, Kataoka H, Satoh Y, et al. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet[C]. Salt Lake City:The IEEE Computer Vision and Pattern Recognition, 2018.
[13]	Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3D convolutional networks[C]. Santiago: IEEE International Conference on Computer Vision, 2015.
[14]	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]. Nevada:The IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[15]	Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017,39(6):1137-1149.
[16]	Neubeck A, Gool L. Efficient non-maximum suppression[C]. Hong Kong:The Eighteenth International Conference on Pattern Recognition, 2006.
[17]	董海青, 林宝军, 刘迎春. 一种有效掏运动目标周围“空洞”的算法[J]. 电子设计工程, 2018,26(18):25-29.
	Dong Haiqing, Lin Baojun, Liu Yingchun. An algorithm to effectively suppress "hole"around moring objects[J]. Electronic Design Engineering, 2018,26(18):25-29.

算法	单轮训练时间	FPS	mAP@0.5/%
R-C3D	1.13 h	389	67.5
Proposed method	0.61 h	722	68.7

IoU AP/mAP	0.3	0.4	0.5	0.6
切割厚板	80.5	79.3	78.0	71.6
清理厚板	69.5	71.2	70.2	54.3
搬运厚板	81.0	80.4	79.5	69.2
厚板装车	49.9	47.2	46.9	45.1
mAP	70.2	69.8	68.7	60.1

基于时序行为检测的工作流识别

Workflow Recognition Based on Temporal Action Detection

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 17

相关文章 4

Metrics

本文评价

推荐阅读 10

[1]	张莹,刘子龙,万伟. 基于Faster R-CNN的无人机车辆目标检测[J]. 电子科技, 2021, 34(11): 11-20.
[2]	邬倩,吴飞,骆立志. 基于深度学习的人体骨架动作识别[J]. 电子科技, 2020, 33(11): 79-83.
[3]	侯严明,李菲菲,陈虬. 基于多特征融合的视频检索算法[J]. 电子科技, 2019, 32(5): 44-49.
[4]	郭桓宇，侯悦民，李康. RFID定位方法及其在智能制造中的应用[J]. , 2017, 30(4): 115-.