一种LSTM与CNN相结合的步态识别方法

doi:10.19665/j.issn1001-2400.2021.05.010

摘要/Abstract

摘要：

针对行人运动过程中拍摄视角、外观变化等因素对步态识别的影响,提出一种长短时记忆网络与卷积神经网络相结合的步态识别方法。该方法首先使用人体三维姿态估计直接获得人体关节的三维坐标;然后根据三维空间中关节之间的周期性运动约束关系,从时间和空间两个维度构建视角鲁棒的三维步态约束模型。其中,模型中的运动约束矩阵用于表征关节运动及人体结构的时序约束特征;动作特征矩阵用于表征关节位移的空间约束特征。接着针对所构建的三维步态约束模型的特点,设计了长短时记忆网络与卷积神经网络相结合的并行网络,学习运动约束矩阵和动作特征矩阵中的步态时空特征。最后,在多视角步态数据库CASIA-B中对网络模型进行了评估。实验结果表明,该方法的识别率高于一些经典的方法,而且在视角变化较大的情况下,识别率没有显著下降,说明构建的步态模型对视角变化具有较高的鲁棒性。

关键词: 深度学习, 步态识别, 人体姿态, 姿态特征矩阵

Abstract:

To solve the problem of influence factors such as view angle and other external factors of variation on gait recognition,we propose a novel and practical gait recognition method combining Long Short Term Memory and Convolutional Neural Networks.Focusing on the three-dimensionality of gait,the new method uses human three-dimensional (3D) pose estimation to obtain 3D coordinates of joints.Then,by analyzing the periodic motion constraint relationships between joints in 3D space,a robust 3D gait constraint model is designed from time and space dimensions.In the model,the motion constraint matrix characterizes both the temporal constraint relationships between joint motion and human body structure,while the action feature matrix characterizes the spatial constraint relationships of the joint position.In addition,based on the characteristics of the 3D gait constraint model,a parallel deep gait recognition network consisting of Long Short Term Memory and Convolutional Neural Networks is developed to extract spatiotemporal features of the model.Finally,the proposed method is evaluated on multi-view gait database CASIA-B.Experimental results show that the recognition rate of the new method is higher than that of some classic methods.At the same time,the recognition rate does not decrease significantly in the case of great view angle changes,illustrating that our method has a state-of-the-art performance and is robust to view changes.

Key words: deep learning, gait recognition, human body pose, pose feature matrix

中图分类号:

TP391

戚艳军,孔月萍,王佳婧,朱旭东. 一种LSTM与CNN相结合的步态识别方法[J]. 西安电子科技大学学报, 2021, 48(5): 78-85.

QI Yanjun,KONG Yueping,WANG Jiajing,ZHU Xudong. Gait recognition method combining LSTM and CNN[J]. Journal of Xidian University, 2021, 48(5): 78-85.

图/表 10

图1

图2

图3

图4

图5

表1

表2

表3

表4

表5

参考文献 21

[1]	王科俊, 丁欣楠, 邢向磊, 等. 多视角步态识别综述[J]. 自动化学报, 2019, 45(5):841-852.
	WANG Kejun, DING Xinnan, XING Xianglei, et al. A Survey of Multi-view Gait Recognition[J]. Acta Automatica Sinica, 2019, 45(5):841-852.
[2]	LU J W, WANG G, MOULIN P. Human Identity and Gender Recognition from Gait Sequences with Arbitrary Walking Directions[J]. IEEE Transactions on Information Forensics and Security, 2014, 9(1):51-61. doi: 10.1109/TIFS.2013.2291969
[3]	GOFFREDO M, BOUCHRIKA I, CARTER J N, et al. Selfcalibrating View-invariant Gait Biometrics[J]. IEEE Transactions on Systems,Man,and Cybernetics: Part B (Cybernetics), 2010, 40(4):997-1008. doi: 10.1109/TSMCB.2009.2031091
[4]	DARWISH S M. Design of Adaptive Biometric Gait Recognition Algorithm with Free Walking Directions[J]. IET Biometrics, 2017, 6(2):53-60. doi: 10.1049/bme2.v6.2
[5]	高莉俊, 汪步云, 许德章. 基于步态子空间的无监督双足步行状态辨识[J]. 模式识别与人工智能, 2018, 31(8):763-772.
	GAO Lijun, WANG Buyun, XU Dezhang. Unsupervised Bipedal Gait Identification Based on Gait Subspace[J]. Pattern Recognition and Artificial Intelligence, 2018, 31(8):763-772.
[6]	KUSAKUNNIRAN W, WU Q, ZHANG J, et al. Gait Recognition Under Various Viewing Angles Based on Correlated Motion Regression[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(6):966-980. doi: 10.1109/TCSVT.2012.2186744
[7]	HU M D, WANG Y H, ZHANG Z X. Cross-view Gait Recognition with Short Probe Sequences:From View Transformation Model to View-independent Stance-independent Identity Vector[J]. International Journal of Pattern Recognition and Artifcial Intelligence, 2013, 27(6):1350017.
[8]	HUYNH-THE T, HUA C H, TU N A. Learning 3D Spatiotemporal Gait Feature by Convolutional Network for Person Identification[J]. Neurocomputing, 2020, 397:192-202. doi: 10.1016/j.neucom.2020.02.048
[9]	冯世灵, 王修晖. 结合非局部与分块特征的跨视角步态识别[J]. 模式识别与人工智能, 2019, 32(9):821-827.
	FENG Shiling, WANG Xiuhui. Cross-View Gait Recognition Combined with Non-local and Part-level Features[J]. Pattern Recognition and Artificial Intelligence, 2019, 32(9):821-827.
[10]	卢来, 邓文, 吴卫祖. 基于改进深度卷积神经网络的步态识别算法[J]. 电子测量与仪器学报, 2019, 33(2):88-93.
	LU Lai, DENG Wen, WU Weizu. Gait Recognition Algorithm Based on Improved Deep Convolution Neural Network[J]. Journal of Electronic Measurement and Instrumentation, 2019, 33(2):88-93.
[11]	WOLF T, BABAEE M, RIGOLL G. Multi-view GAIT Recognition Using 3D Convolutional Neural Networks[C]// Proceedings of the IEEE International Conference on Image Processing.Piscataway:IEEE, 2016:4165-4169.
[12]	YU S Q, CHEN H F, REYES E G, et al. GaitGAN:Invariant Gait Feature Extraction Using Generative Adversarial Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:30-37.
[13]	汪涛, 汪泓章, 夏懿, 等. 基于卷积神经网络与注意力模型的人体步态识别[J]. 传感技术学报, 2019, 32(7):1027-1033.
	WANG Tao, WANG Hongzhang, XIA Yi, et al. Human Gait Recognition Based on Convolutional Neural Network and Attention Model[J]. Chinese Journal of Sensors and Actuators, 2019, 32(7):1027-1033.
[14]	L'OPEZ-FERN'ANDEZ D, MADRID-CUEVAS F J, CARMONA-POYATO A, et al. A New Approach for Multi-view Gait Recognition on Unconstrained Paths[J]. Journal of Visual Communication and Image Representation, 2016(38):396-406.
[15]	TANG J, LUO J, TJAHJADI T, et al. 2.5D Multi-view Gait Recognition based on Point Cloud Registration[J]. Sensors, 2014, 14(4):6124-6143. doi: 10.3390/s140406124
[16]	李健, 张皓若, 何斌. 结合形变模型的人体姿态估计优化算法[J]. 西安电子科技大学学报, 2020, 47(2):23-31.
	LI Jian, ZHANG Haoruo, HE Bin. Optimization Algorithm for Estimating the Human Pose by Using the Morphable Model[J]. Journal of Xidian University, 2020, 47(2):23-31.
[17]	LIAO R J, CAO C S, GARCIA E B, et al. Pose-Based Temporal-Spatial Network(PTSN) for Gait Recognition with Carrying and Clothing Variations[C]// Proceedings of the Chinese Conference on Biometric Recognition.Berlin:Springer, 2017:474-483.
[18]	FENG Y, LI Y C, LUO J B. Learning Effective Gait Features Using LSTM[C]// Proceedings of the 23rd IEEE International Conference on Pattern Recognition.Piscataway:IEEE, 2016: 16835724.
[19]	LIAO R, YU S, AN W, et al. A Model-based Gait Recognition Method with Body Pose and Human Prior Knowledge[J]. Pattern Recognition, 2020 (98):107069.
[20]	YU S Q, CHEN H F, WANG Q, et al. Invariant Feature Extraction for Gait Recognition Using Only one Uniform Model[J]. Neurocomputing, 2017 (239):81-93.
[21]	YU S Q, LIAO R J, AN W Z, et al. GaitGANv2:Invariant Gait Feature Extraction Using Generative Adversarial Networks[J]. Pattern Recognition, 2019 (87):179-189.

LSTM分支结构及参数		CNN分支结构及参数
层名	参数	层名	卷积核数量	卷积核尺寸	步长	激活函数
LSTM-1	输入维数:44 输出维数:50 激活函数:Tanh	Con-1	32	2×2	1	ReLU
		Pool-1		2×2	2
		Con-2	64	2×2	1	ReLU
		Pool-2		2×2	2
LSTM-2	输入维数:50 输出维数:64 激活函数:Tanh	Con-3	128	2×2	1	ReLU
		Pool-3		2×2	2
		Con-4	256	2×2	1	ReLU
		Pool-4		2×2	2
Flatten	3 200维	Flatten	3 072维

注册集视角	验证集视角
注册集视角	0°	18°	36°	54°	72°	90°	108°	126°	144°	162°	180°
0°	91.25	79.20	62.83	51.27	29.24	33.10	37.43	29.20	36.60	48.30	63.00
18°	85.71	94.04	88.75	81.26	59.02	78.50	39.10	42.71	37.58	72.94	62.19
36°	74.86	82.16	95.48	90.63	84.43	79.25	69.93	56.85	40.29	57.76	51.45
54°	40.20	76.72	86.26	91.54	87.12	76.41	75.80	54.30	65.91	43.38	30.87
72°	32.17	54.07	81.58	84.09	95.25	91.52	77.46	81.36	65.87	38.24	25.20
90°	25.12	50.21	73.42	90.35	90.47	96.28	88.66	84.03	75.75	41.53	25.17
108°	32.03	66.44	62.38	80.34	92.58	92.17	96.02	95.65	84.19	61.94	43.20
126°	43.11	43.40	59.26	72.87	76.32	88.20	86.42	92.86	79.83	70.93	50.69
144°	57.65	52.94	55.02	68.69	82.76	82.47	77.49	87.17	95.26	88.06	76.96
162°	60.46	63.35	47.90	65.47	52.68	68.22	54.33	73.28	80.90	92.36	79.05
180°	68.75	67.31	48.01	27.83	30.57	28.84	23.36	33.72	65.74	83.47	93.82

行走状态 (注册集-验证集)	验证集视角
行走状态 (注册集-验证集)	0°	18°	36°	54°	72°	90°	108°	126°	144°	162°	180°
nm-nm	55.57	69.14	69.17	73.12	70.95	74.09	66.00	66.47	70.08	63.54	54.69
nm-bg	38.02	43.43	51.69	46.01	52.93	43.07	49.60	51.06	52.87	40.30	36.12
nm-cl	20.83	25.90	40.20	37.50	35.84	33.97	40.69	37.04	36.56	33.71	26.16

方法	行走状态(注册集-验证集)
方法	nm- nm	nm- bg	nm- cl
LC-POSEGAIT	66.62	45.92	33.49
LSTM分支	62.25	42.70	30.62
CNN分支	50.27	34.83	27.40

方法	行走状态(注册集-验证集)
方法	nm- nm	nm- bg	nm- cl
PoseGait^[19]	63.78	42.52	31.98
SPAE^[20]	62.82	40.38	26.05
GaitGANv2^[21]	66.34	46.17	25.91
新方法	66.62	45.92	33.49