一种LSTM与CNN相结合的步态识别方法

doi:10.19665/j.issn1001-2400.2021.05.010

Abstract

Abstract:

To solve the problem of influence factors such as view angle and other external factors of variation on gait recognition,we propose a novel and practical gait recognition method combining Long Short Term Memory and Convolutional Neural Networks.Focusing on the three-dimensionality of gait,the new method uses human three-dimensional (3D) pose estimation to obtain 3D coordinates of joints.Then,by analyzing the periodic motion constraint relationships between joints in 3D space,a robust 3D gait constraint model is designed from time and space dimensions.In the model,the motion constraint matrix characterizes both the temporal constraint relationships between joint motion and human body structure,while the action feature matrix characterizes the spatial constraint relationships of the joint position.In addition,based on the characteristics of the 3D gait constraint model,a parallel deep gait recognition network consisting of Long Short Term Memory and Convolutional Neural Networks is developed to extract spatiotemporal features of the model.Finally,the proposed method is evaluated on multi-view gait database CASIA-B.Experimental results show that the recognition rate of the new method is higher than that of some classic methods.At the same time,the recognition rate does not decrease significantly in the case of great view angle changes,illustrating that our method has a state-of-the-art performance and is robust to view changes.

Key words: deep learning, gait recognition, human body pose, pose feature matrix

CLC Number:

TP391

QI Yanjun,KONG Yueping,WANG Jiajing,ZHU Xudong. Gait recognition method combining LSTM and CNN[J].Journal of Xidian University, 2021, 48(5): 78-85.

Figures/Tables 10

References 21

[1]	王科俊, 丁欣楠, 邢向磊, 等. 多视角步态识别综述[J]. 自动化学报, 2019, 45(5):841-852.
	WANG Kejun, DING Xinnan, XING Xianglei, et al. A Survey of Multi-view Gait Recognition[J]. Acta Automatica Sinica, 2019, 45(5):841-852.
[2]	LU J W, WANG G, MOULIN P. Human Identity and Gender Recognition from Gait Sequences with Arbitrary Walking Directions[J]. IEEE Transactions on Information Forensics and Security, 2014, 9(1):51-61. doi: 10.1109/TIFS.2013.2291969
[3]	GOFFREDO M, BOUCHRIKA I, CARTER J N, et al. Selfcalibrating View-invariant Gait Biometrics[J]. IEEE Transactions on Systems,Man,and Cybernetics: Part B (Cybernetics), 2010, 40(4):997-1008. doi: 10.1109/TSMCB.2009.2031091
[4]	DARWISH S M. Design of Adaptive Biometric Gait Recognition Algorithm with Free Walking Directions[J]. IET Biometrics, 2017, 6(2):53-60. doi: 10.1049/bme2.v6.2
[5]	高莉俊, 汪步云, 许德章. 基于步态子空间的无监督双足步行状态辨识[J]. 模式识别与人工智能, 2018, 31(8):763-772.
	GAO Lijun, WANG Buyun, XU Dezhang. Unsupervised Bipedal Gait Identification Based on Gait Subspace[J]. Pattern Recognition and Artificial Intelligence, 2018, 31(8):763-772.
[6]	KUSAKUNNIRAN W, WU Q, ZHANG J, et al. Gait Recognition Under Various Viewing Angles Based on Correlated Motion Regression[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(6):966-980. doi: 10.1109/TCSVT.2012.2186744
[7]	HU M D, WANG Y H, ZHANG Z X. Cross-view Gait Recognition with Short Probe Sequences:From View Transformation Model to View-independent Stance-independent Identity Vector[J]. International Journal of Pattern Recognition and Artifcial Intelligence, 2013, 27(6):1350017.
[8]	HUYNH-THE T, HUA C H, TU N A. Learning 3D Spatiotemporal Gait Feature by Convolutional Network for Person Identification[J]. Neurocomputing, 2020, 397:192-202. doi: 10.1016/j.neucom.2020.02.048
[9]	冯世灵, 王修晖. 结合非局部与分块特征的跨视角步态识别[J]. 模式识别与人工智能, 2019, 32(9):821-827.
	FENG Shiling, WANG Xiuhui. Cross-View Gait Recognition Combined with Non-local and Part-level Features[J]. Pattern Recognition and Artificial Intelligence, 2019, 32(9):821-827.
[10]	卢来, 邓文, 吴卫祖. 基于改进深度卷积神经网络的步态识别算法[J]. 电子测量与仪器学报, 2019, 33(2):88-93.
	LU Lai, DENG Wen, WU Weizu. Gait Recognition Algorithm Based on Improved Deep Convolution Neural Network[J]. Journal of Electronic Measurement and Instrumentation, 2019, 33(2):88-93.
[11]	WOLF T, BABAEE M, RIGOLL G. Multi-view GAIT Recognition Using 3D Convolutional Neural Networks[C]// Proceedings of the IEEE International Conference on Image Processing.Piscataway:IEEE, 2016:4165-4169.
[12]	YU S Q, CHEN H F, REYES E G, et al. GaitGAN:Invariant Gait Feature Extraction Using Generative Adversarial Networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2017:30-37.
[13]	汪涛, 汪泓章, 夏懿, 等. 基于卷积神经网络与注意力模型的人体步态识别[J]. 传感技术学报, 2019, 32(7):1027-1033.
	WANG Tao, WANG Hongzhang, XIA Yi, et al. Human Gait Recognition Based on Convolutional Neural Network and Attention Model[J]. Chinese Journal of Sensors and Actuators, 2019, 32(7):1027-1033.
[14]	L'OPEZ-FERN'ANDEZ D, MADRID-CUEVAS F J, CARMONA-POYATO A, et al. A New Approach for Multi-view Gait Recognition on Unconstrained Paths[J]. Journal of Visual Communication and Image Representation, 2016(38):396-406.
[15]	TANG J, LUO J, TJAHJADI T, et al. 2.5D Multi-view Gait Recognition based on Point Cloud Registration[J]. Sensors, 2014, 14(4):6124-6143. doi: 10.3390/s140406124
[16]	李健, 张皓若, 何斌. 结合形变模型的人体姿态估计优化算法[J]. 西安电子科技大学学报, 2020, 47(2):23-31.
	LI Jian, ZHANG Haoruo, HE Bin. Optimization Algorithm for Estimating the Human Pose by Using the Morphable Model[J]. Journal of Xidian University, 2020, 47(2):23-31.
[17]	LIAO R J, CAO C S, GARCIA E B, et al. Pose-Based Temporal-Spatial Network(PTSN) for Gait Recognition with Carrying and Clothing Variations[C]// Proceedings of the Chinese Conference on Biometric Recognition.Berlin:Springer, 2017:474-483.
[18]	FENG Y, LI Y C, LUO J B. Learning Effective Gait Features Using LSTM[C]// Proceedings of the 23rd IEEE International Conference on Pattern Recognition.Piscataway:IEEE, 2016: 16835724.
[19]	LIAO R, YU S, AN W, et al. A Model-based Gait Recognition Method with Body Pose and Human Prior Knowledge[J]. Pattern Recognition, 2020 (98):107069.
[20]	YU S Q, CHEN H F, WANG Q, et al. Invariant Feature Extraction for Gait Recognition Using Only one Uniform Model[J]. Neurocomputing, 2017 (239):81-93.
[21]	YU S Q, LIAO R J, AN W Z, et al. GaitGANv2:Invariant Gait Feature Extraction Using Generative Adversarial Networks[J]. Pattern Recognition, 2019 (87):179-189.

LSTM分支结构及参数		CNN分支结构及参数
层名	参数	层名	卷积核数量	卷积核尺寸	步长	激活函数
LSTM-1	输入维数:44 输出维数:50 激活函数:Tanh	Con-1	32	2×2	1	ReLU
		Pool-1		2×2	2
		Con-2	64	2×2	1	ReLU
		Pool-2		2×2	2
LSTM-2	输入维数:50 输出维数:64 激活函数:Tanh	Con-3	128	2×2	1	ReLU
		Pool-3		2×2	2
		Con-4	256	2×2	1	ReLU
		Pool-4		2×2	2
Flatten	3 200维	Flatten	3 072维

注册集视角	验证集视角
注册集视角	0°	18°	36°	54°	72°	90°	108°	126°	144°	162°	180°
0°	91.25	79.20	62.83	51.27	29.24	33.10	37.43	29.20	36.60	48.30	63.00
18°	85.71	94.04	88.75	81.26	59.02	78.50	39.10	42.71	37.58	72.94	62.19
36°	74.86	82.16	95.48	90.63	84.43	79.25	69.93	56.85	40.29	57.76	51.45
54°	40.20	76.72	86.26	91.54	87.12	76.41	75.80	54.30	65.91	43.38	30.87
72°	32.17	54.07	81.58	84.09	95.25	91.52	77.46	81.36	65.87	38.24	25.20
90°	25.12	50.21	73.42	90.35	90.47	96.28	88.66	84.03	75.75	41.53	25.17
108°	32.03	66.44	62.38	80.34	92.58	92.17	96.02	95.65	84.19	61.94	43.20
126°	43.11	43.40	59.26	72.87	76.32	88.20	86.42	92.86	79.83	70.93	50.69
144°	57.65	52.94	55.02	68.69	82.76	82.47	77.49	87.17	95.26	88.06	76.96
162°	60.46	63.35	47.90	65.47	52.68	68.22	54.33	73.28	80.90	92.36	79.05
180°	68.75	67.31	48.01	27.83	30.57	28.84	23.36	33.72	65.74	83.47	93.82

行走状态 (注册集-验证集)	验证集视角
行走状态 (注册集-验证集)	0°	18°	36°	54°	72°	90°	108°	126°	144°	162°	180°
nm-nm	55.57	69.14	69.17	73.12	70.95	74.09	66.00	66.47	70.08	63.54	54.69
nm-bg	38.02	43.43	51.69	46.01	52.93	43.07	49.60	51.06	52.87	40.30	36.12
nm-cl	20.83	25.90	40.20	37.50	35.84	33.97	40.69	37.04	36.56	33.71	26.16

方法	行走状态(注册集-验证集)
方法	nm- nm	nm- bg	nm- cl
LC-POSEGAIT	66.62	45.92	33.49
LSTM分支	62.25	42.70	30.62
CNN分支	50.27	34.83	27.40

方法	行走状态(注册集-验证集)
方法	nm- nm	nm- bg	nm- cl
PoseGait^[19]	63.78	42.52	31.98
SPAE^[20]	62.82	40.38	26.05
GaitGANv2^[21]	66.34	46.17	25.91
新方法	66.62	45.92	33.49

Gait recognition method combining LSTM and CNN

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 21

Related Articles 15

Metrics

Comments

Recommended 10

[1]	LIU Jiawei,ZHANG Wenhui,KOU Xiaoli,LI Yanni. Harnessing adversarial examples via input denoising and hidden information restoring [J]. Journal of Xidian University, 2021, 48(6): 23-31.
[2]	LI Peng,FENG Cunqian,XU Xuguang,TANG Zixiang. Ballistic target fretting classification network based on Bayesian optimization [J]. Journal of Xidian University, 2021, 48(5): 139-148.
[3]	YAN Jia,CAO Yudong,REN Jiaxing,CHEN Donghao,LI Xiaohui. Deep asymmetric compression Hashing algorithm [J]. Journal of Xidian University, 2021, 48(5): 212-221.
[4]	NING Yang,DU Jianchao,HAN Shuo,YANG Chuankai. Fire segmentation based on the improved DeeplabV3+ and the analytical method for fire development [J]. Journal of Xidian University, 2021, 48(5): 38-46.
[5]	ZHOU Peng,YANG Jun. Semantic segmentation of remote sensing images based on neural architecture search [J]. Journal of Xidian University, 2021, 48(5): 47-57.
[6]	SONG Jianfeng,MIAO Qiguang,WANG Chongxiao,XU Hao,YANG Jin. Multi-scale single object tracking based on the attention mechanism [J]. Journal of Xidian University, 2021, 48(5): 110-116.
[7]	ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network [J]. Journal of Xidian University, 2021, 48(5): 15-22.
[8]	XU Huanhuan,LI Hongmei,LI Fuyu,SUN Xuemei. Gait recognition method based on spatial-temporal convolution [J]. Journal of Xidian University, 2021, 48(4): 144-150.
[9]	HUI Haisheng,ZHANG Xueying,WU Zelin,LI Fenglian. Method for stroke lesion segmentation using the primary-auxiliary path attention compensation network [J]. Journal of Xidian University, 2021, 48(4): 200-208.
[10]	SUN Haojie,LI Miaoyu,ZHANG Panpan,XU Pengfei. Self-supervised facial asymmetry learning for automatic evaluation of facial paralysis [J]. Journal of Xidian University, 2021, 48(3): 115-122.
[11]	ZHANG Hua,GAO Haoran,YANG Xingguo,LI Wenmin,GAO Fei,WEN Qiaoyan. TargetedFool:an algorithm for achieving targeted attacks [J]. Journal of Xidian University, 2021, 48(1): 149-159.
[12]	YANG Hongyu,ZENG Renyun. Method for assessment of network security situation with deep learning [J]. Journal of Xidian University, 2021, 48(1): 183-190.
[13]	ZHANG Lu,SUN Rong,LIU Jingwei. Cloned piggybacking framework for distributed storage [J]. Journal of Xidian University, 2020, 47(6): 139-147.
[14]	HU Jianwei,ZHAO Wei,CUI Yanpeng,CUI Junjie. PHP code vulnerability mining technology based on theimproved ASTNN [J]. Journal of Xidian University, 2020, 47(6): 164-173.
[15]	GUO Liujun,ZHANG Xueying,CHEN Guijun. Deep linear discriminant analysis for two-stage brain-controlled character spelling decoding [J]. Journal of Xidian University, 2020, 47(4): 109-116.