基于姿态估计的八段锦序列动作识别与评估

doi:10.16180/j.cnki.issn1007-7820.2022.12.012

摘要/Abstract

摘要：

动作评估与反馈可有效辅助健身运动练习者提高锻炼收益。为了实现八段锦动作的自动量化评估,文中提出一种人体序列动作识别与评估方法。采用姿态估计算法OpenPose提取人体关键点坐标并进行归一化,剔除冗余点。根据动作特点构造出融合关键点位置、距离、关节角度和关键点速度的特征向量,通过多层感知机训练出动作分类器模型。所提方法在KTH和自制八段锦数据集上的动作识别准确率分别达到96.7%和98.7%。基于八段锦动作识别结果构建动作序列,采用动态时间规整算法计算两组八段锦动作序列的相似度,对比实验结果表明该相似度可有效评估动作的完整性及同步性。

关键词: 姿态估计, OpenPose, 动作识别, 特征提取, 特征融合, 动态时间规整, 八段锦, 动作评估

Abstract:

Action evaluation and feedback can assist fitness exercisers to improve exercise benefits effectively. In order to realize the automatic quantitative evaluation of eight-section brocade movement, a method of recognition and evaluation of human body sequence movements is proposed. The pose estimation algorithm OpenPose is used to extract the coordinates of the key points of the human body, and then normalize them and eliminate redundant points. According to the characteristics of the action, the feature vector of the fusion key points position, distance, joint angle and key points speed is constructed, and the multi-layer perceptron is employed to train the action classification. The accuracy of action recognition on the KTH and self-made eight-section brocade data sets attains to 96.7% and 98.7%, respectively. Based on the eight-section brocade action recognition results, an action sequence is constructed, and the dynamic time warping algorithm is used to calculate the similarity of the two groups of eight-section brocade action sequences. The comparative experimental results show that the similarity can effectively evaluate the integrity and synchronization of actions.

Key words: pose estimation, OpenPose, action recognition, feature extraction, feature fusion, dynamic time warping, eight-section brocade, action evaluation

中图分类号:

TP391

苏波,柴自强,王莉,崔帅华. 基于姿态估计的八段锦序列动作识别与评估[J]. 电子科技, 2022, 35(12): 84-90.

SU Bo,CHAI Ziqiang,WANG Li,CUI Shuaihua. Eight-Section Brocade Sequence Action Recognition and Evaluation Based on Pose Estimation[J]. Electronic Science and Technology, 2022, 35(12): 84-90.

图/表 22

图1

图2

表1

图3

表2

图4

图5

表3

图6

表4

表5

图7

图8

图9

表6

图10

表7

图11

图12

图13

图14

表8

参考文献 18

[1]	Choudhury N A, Moulik S, Choudhury S. Cloud-based real-time and remote human activity recognition system using wearable sensors[C]. Taoyuan: Proceedings of the IEEE International Conference on Consumer Electronics-Taiwan, 2020.
[2]	Tao L. Intelligent recognition method of human motion global features based on kinect skeleton information[C]. Harbin:Proceedings of the IEEE International Conference on Industrial Application of Artificial Intelligence, 2020.
[3]	Ma H, Liu H. Research on human motion recognition system based on MEMS sensor network[C]. Chengdu: Proceedings of IEEE the Fourth Advanced Information Technology, Electronic and Automation Control Conference, 2019.
[4]	Cui M M. Fang J D, Zhao Y D. Emotion recognition of human body's posture in open environment[C]. Hefei: Proceedings of the Chinese Control and Decision Conference, 2020.
[5]	Wei S E, Ramakrishna V, Kanade T, et al. Convolutional pose machines[C]. Las Vegas: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[6]	Batchuluun G, Kang J K, Nguyen D T, et al. Action recognition from thermal videos using joint and skeleton information[J]. IEEE Access, 2021(9):11716-11733.
[7]	Cao Z, Simon T, Wei S E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]. Honolulu: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[8]	Cao Z, Hidalgo G, Simon T, et al. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(1):172-186. doi: 10.1109/TPAMI.2019.2929257
[9]	Fang H S, Xie S, Tai Y W, et al. RMPE: Regional multi-person pose estimation[C]. Venice: Proceedings of the IEEE International Conference on Computer Vision, 2017.
[10]	Chen Y, Wang Z, Peng Y, et al. Cascaded pyramid network for multi-person pose estimation[C]. Salt Lake City: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[11]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[C]. San Diego: Proceedings of the Third International Conference on Learning Representations, 2015.
[12]	Schuldt C, Laptev I, Caputo B. Recognizing human actions: A local SVM approach[C]. Cambridge: Proceedings of the Seventeenth International Conference on Pattern Recognition, 2004.
[13]	Turk M A, Pentland A. Face recognition using eigenfaces[J]. Journal of Cognitive Neuroscience, 2012, 3(1):71-86. doi: 10.1162/jocn.1991.3.1.71
[14]	Megrhi S, Jmal M, Souidene W, et al. Spatio-temporal action localization and detection for human action recognition in big dataset[J]. Journal of Visual Communication and Image Representation, 2016, 4(1):375-390.
[15]	鹿天然, 于凤芹, 陈莹. 一种基于线性序列差异分析降维的人体行为识别方法[J]. 计算机工程, 2019, 45(3):237-241.
	Lu Tianran, Yu Fengqin, Chen Ying. A human action recognition method based on LSDA reduction[J]. Computer Engineering, 2019, 45(3):237-241.
[16]	刘命强. 基于深度学习的人体动作识别研究[D]. 开封: 河南大学, 2018.
	Liu Mingqiang. Research on human action recognition based on deep learning[D]. Kaifeng: Henan University, 2018.
[17]	Sakoe H, Chiba S. Dynamic programming algorithm optimization for spoken word recognition[J]. IEEE Transactions on Acoustics Speech and Signal Processing, 1978, 26(1):43-49. doi: 10.1109/TASSP.1978.1163055
[18]	黄东方, 杨晶东. 基于改进ND-DTW算法的动态手势识别[J]. 电子科技, 2017, 30(3):37-40.
	Huang Dongfang, Yang Jingdong. Dynamic gesture recognition based on improved N-dimensional dynamic time warping[J]. Electronic Science and Technology, 2017, 30(3):37-40.

序号	部位	序号	部位
0	鼻子	9	右膝
1	脖子	10	右踝
2	右肩	11	左髋
3	右肘	12	左膝
4	右手腕	13	左踝
5	左肩	14	右眼
6	左肘	15	左眼
7	左手腕	16	右耳
8	右髋	17	左耳

关节距离L	说明	关节角θ	说明
L₁	d_(1-3)	θ₁	∠(1-2-3)
L₂	d_(2-4)	θ₂	∠(2-3-4)
L₃	d_(1-6)	θ₃	∠(1-5-6)
L₄	d_(5-7)	θ₄	∠(5-6-7)
L₅	d_(1-9)	θ₅	∠(1-8-9)
L₆	d_(8-10)	θ₆	∠(8-9-10)
L₇	d_(1-12)	θ₇	∠(1-11-12)
L₈	d_(11-13)	θ₈	∠(11-12-13)
L₉	d_(2-8)	θ₉	∠(2-1-8)
L₁₀	d_(5-11)	θ₁₀	∠(5-1-11)
L₁₁	d_(8-13)	θ₁₁	∠(8-9-13)
L₁₂	d_(10-11)	θ₁₂	∠(10-12-11)

特征类型	维度大小	说明
关键点位置	N×14×2	14个关键点的归一化坐标
关键点间距	N×12	部分关键点之间的欧式距离
关节角度	N×12	部分关节之间所呈的夹角
关键点速度	(N-1)×14×2	14个关键点x与y方向的速度

组别	特征组合类型	特征初始维数	平均准确率/%
第1组	空间特征	520	87.5
第2组	运动特征	252	71.8
第3组	融合特征	772	94.6

方法	准确率/%
光流+SURF^[14]	94.9
DT+SVM^[15]	95.4
CNN^[16]	92.2
本文(融合特征)	96.7