一种面向时空神经网络的潜在情绪识别方法

doi:10.19665/j.issn1001-2400.2021.04.021

Abstract

Abstract:

(Micro) expression has a certain effect on emotion recognition,but in the case of artificial concealment,it is prone to misjudgment.Although the recognition effect of physiological signals is more accurate,the data collection is often complicated,which is not convenient for rapid personnel emotion checking.In response to the above problem,this paper adopts a non-contact chromatic model-based method to collect pulse signals,extract features based on the pulse signals,and integrate the proposed spatio-temporal neural network to realize potential emotion recognition.Experimental results show that the proposed two-way latent emotion recognition framework can well integrate the emotion information contained in micro-expressions and physiological signals,and that the effect in micro-expression recognition is improved to some extent compared with the current micro-expression recognition algorithms commonly used at this stage.

Key words: latent emotion recognition, chroma model, deeplearning, facial pulse signal, neural networks, decision fusion

CLC Number:

TP183

SONG Jianqiao,WANG Feng,NIU Jin,SHI Zezhou,MA Junhui. Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals[J].Journal of Xidian University, 2021, 48(4): 159-167.

Figures/Tables 12

References 25

[1]	MUKHERJEE S, VAMSHI B, REDDY K V S V K, et al. Recognizing Facial Expressions Using Novel Motion Based Features[C]//Proceedings of the Tenth Indian Conference on Computer Vision,Graphics and Image Processing.Piscataway:IEEE, 2016: 32.
[2]	LI X, HONG X, MOILANEN A, et al. Towards Reading Hidden Emotions:A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods[J]. IEEE Transactions on Affective Computing, 2018, 9(4):563-577. doi: 10.1109/T-AFFC.5165369
[3]	WU Q, SHEN X B, FU X L. The Machine Knows What You Are Hiding:An Automatic Micro-Expression Recognition System[C]//Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction-Volume Part II.Springer Berlin, 2011:152-162.
[4]	LU H, KPALMA K, RONSIN J. Motion Descriptors for Micro-Expression Recognition[J]. Signal Processing-Image Communication, 2018, 67:108-117. doi: 10.1016/j.image.2018.05.014
[5]	LIU Y J, ZHANG J K, YAN W J, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/T-AFFC.5165369
[6]	LI Q Y, YU J, KURIHARA T, et al. Micro-Expression Analysis by Fusing Deep Convolutional Neural Network and Optical Flow[C]//Proceedings of the 2018 5th International Conference on Control,Decision and Information Technologies.Piscataway:IEEE, 2018:265-270.
[7]	PENG M, WU Z, ZHANG Z H, et al. From Macro to Micro Expression Recognition:Deep Learning on Small Datasets Using Transfer Learning[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 2018:657-661.
[8]	WANG S J, LI B J, LIU Y J, et al. Micro-Expression Recognition with Small Sample Size by Transferring Long-Term Convolutional Neural Network[J]. Neurocomputing, 2018, 312:251-262. doi: 10.1016/j.neucom.2018.05.107
[9]	KIM D H, BADDAR W J, RO Y M. Micro-expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations[C]//Proceedings of the 2016 ACM Multimedia Conference.Amsterdam:ACM, 2016:382-386.
[10]	KIM D H, BADDAR W, JANG J, et al. Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression Recognition[J]. IEEE Transactions on Affective Computing, 2019, 10(2):223-236. doi: 10.1109/T-AFFC.5165369
[11]	KHOR H Q, SEE J, PHAN R C W, et al. Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 2018:667-674.
[12]	FRANTZIDIS C A, BRATSAS C, PAPADELIS C L, et al. Toward Emotion Aware Computing:an Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli[J]. IEEE Transactions on Information Technology in Biomedicine, 2010, 14(3):589-597. doi: 10.1109/TITB.2010.2041553
[13]	KHEZRI M, FIROOZABADI M, SHARAFAT A R. Reliable Emotion Recognition System Based on Dynamic Adaptive Fusion of Forehead Biopotentials and Physiological Signals[J]. Computer Methods and Programs in Biomedicine, 2015, 122(2):149-164. doi: 10.1016/j.cmpb.2015.07.006
[14]	TORRES-VALENCIA C, ALVAREZ-LOPEZ M, OROZCO-GUTIERREZ A. SVM-Based Feature Selection Methods for Emotion Recognition from Multimodal Data[J]. Journal on Multimodal User Interfaces, 2017, 11(1):9-23. doi: 10.1007/s12193-016-0222-y
[15]	DUC B, BIGUN E S, BIGUN J, et al. Fusion of Audio and Video Information for Multi Modal Person Authentication[J]. Pattern Recognition Letters, 1997, 18(9):835-843. doi: 10.1016/S0167-8655(97)00071-8
[16]	BAILENSON J N, PONTIKAKIS E D, MAUSS I B, et al. Real-Time Classification of Evoked Emotions Using Facial Feature Tracking and Physiological Responses[J]. International Journal of Human Computer Studies, 2008, 66(5):303-317. doi: 10.1016/j.ijhcs.2007.10.011
[17]	TRIPATHI S, BEIGI H. Multi-modal Emotion Recognition on IEMOCAP Dataset Using Deep Learning[J/OL].[2020-03-20].https://arxiv.org/pdf/1804.05788.pdf .
[18]	ZHANG D, WU L, LI S, et al. Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions[C]//Proceedings of the 2019 IEEE International Conference on Multimedia and Expo.Piscataway:IEEE Computer Society, 2019:724-729.
[19]	VIOLA P, JONES M. Robust Real-Time Object Detection[J]. International Journal of Computer Vision, 2001, 57:137-154. doi: 10.1023/B:VISI.0000013087.49260.fb
[20]	LIBERZON D, TEMPO R. Common Lyapunov Functions and Gradient Algorithms[J]. IEEE Transactions on Automatic Control, 2004, 49(6):990-994. doi: 10.1109/TAC.2004.829632
[21]	COSTA M, GOLDBERGER A, PENG C K. Multiscale Entropy Analysis of Biological Signals[J]. Physical Review E, 2005, 71(2):21906. doi: 10.1103/PhysRevE.71.021906
[22]	NITISH S, HINTON G, KRIZHEVSKY A, et al. Dropout:a Simple Way to Prevent Neural Networks from Overfitting[J]. Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[23]	HUANG X H, ZHAO G Y, HONG X P, et al. Spontaneous Facial Micro-Expression Analysis Using Spatiotemporal Completed Local Quantized Patterns[J]. Neurocomputing, 2016, 175:564-578. doi: 10.1016/j.neucom.2015.10.096
[24]	LI J, WANG Y D, JOHN S, et al. Micro-Expression Recognition Based on 3D Flow Convolutional Neural Network[J]. Pattern Analysis and Applications, 2019, 22(4):1331-1339. doi: 10.1007/s10044-018-0757-5
[25]	LIU Y J, ZHANG J K, YAN W J, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/T-AFFC.5165369

网络层名称	卷积核个数	卷积核尺寸	步长	输出维度
输入				64×64×96
Conv3D_1	32	3×3×15	1×1×1	32×62×62×82
Conv3D_2	32	3×3×15	1×1×1	32×60×60×68
Maxpooling3D		3×3×3	1×1×1	32×20×20×22
ConvLSTM_1	16	3×3	1×1×1	32×16×18×20
ConvLSTM_2	16	3×3	1×1×1	32×16×16×18
Flatten				147 456
FC_1				128

情绪	CAS(ME)^2	USPE
生气	275=220+55	515=412+103
厌恶	370=296+74	435=348+87
开心	655=524+131	600=480+120
惊讶	180=144+36
悲伤		490=392+98
总计	1 480=1 184+296	2 040=1 632+408

卷积核尺寸		不同尺寸3D卷积核的识别率/%
Conv3D_1	Conv3D_2	CAS(ME)^2	TYUTPE
3×3×7	3×3×7	69.83	70.26
3×3×15	3×3×15	75.78	73.92
3×3×19	3×3×19	46.36	51.23
5×5×7	5×5×7	69.49	70.52
5×5×15	5×5×15	68.67	67.31
5×5×19	5×5×19	48.92	47.36

卷积核尺寸		不同尺寸3D卷积核的识别率/%
ConvLSTM_1	ConvLSTM_2	CAS(ME)^2	TYUTPE
2×2	2×2	69.95	71.01
3×3	3×3	77.02	74.83
4×4	4×4	56.20	58.35
5×5	5×5	54.73	60.53

实验	CAS(ME)^2数据库情绪识别率/%					USPE数据库情绪识别率/%
实验	生气	厌恶	开心	惊讶	平均	生气	厌恶	开心	悲伤	平均
实验1	58.12	59.45	67.76	71.93	64.31	57.88	59.10	75.96	62.90	64.04
实验2	67.52	76.94	83.23	80.39	77.02	73.45	78.62	81.83	65.42	74.83
实验3	73.20	74.13	81.58	85.43	78.59	68.59	76.51	88.62	73.93	76.91

Potential emotion recognition based on the fusion of the spatio-temporal neural network and facial pulse signals

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 25

Related Articles 15

Metrics

Comments

Recommended 10

方法	CAS(ME)^2识别率	USPE识别率
LBP-TOP	68.46	66.59
STCLQP^[23]	69.02	67.30
3D-FCNN^[24]	72.06	71.36
MDMO^[25]	65.23	63.35
CNN with Augmentation^[17]	74.32	72.11
文中方法	78.59	76.91

[1]	LV Wenkai,YANG Pengfei,DING Yunqing,ZHANG Heyu,ZHENG Tianyang. JEDERL:A task scheduling optimization algorithm for heterogeneous computing platforms [J]. Journal of Xidian University, 2021, 48(6): 67-74.
[2]	ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network [J]. Journal of Xidian University, 2021, 48(5): 15-22.
[3]	HUI Haisheng,ZHANG Xueying,WU Zelin,LI Fenglian. Method for stroke lesion segmentation using the primary-auxiliary path attention compensation network [J]. Journal of Xidian University, 2021, 48(4): 200-208.
[4]	CAO Yi,CAI Xiaodong. Effective learning strategy for hard samples [J]. Journal of Xidian University, 2021, 48(3): 99-105.
[5]	WANG Ping,JIANG Yuze,ZHAO Guanghui. Object detection based on the multiscale location Enhancement network [J]. Journal of Xidian University, 2021, 48(3): 85-90.
[6]	GUO Zekun,TIAN Long,HAN Ning,WANG Penghui,LIU Hongwei,CHEN Bo. Radar HRRP based few-shot target recognition with CNN-SSD [J]. Journal of Xidian University, 2021, 48(2): 7-14.
[7]	ZHANG Shudong,GAO Haichang,CAO Xiwen,KANG Shuai. Adaptive fast and targeted adversarial attack for speech recognition [J]. Journal of Xidian University, 2021, 48(1): 168-175.
[8]	DANG Jisheng,YANG Jun. 3D model recognition and segmentation based on multi-feature fusion [J]. Journal of Xidian University, 2020, 47(4): 149-157.
[9]	LI Kunlun,ZHANG Lu,XU Hongke,SONG Huansheng. Waveletdomain dilated network for fast low-dose CT image reconstruction [J]. Journal of Xidian University, 2020, 47(4): 86-93.
[10]	NGUYEN Van-Truong,CAI Jueping,WEI Linyu,CHU Jie. Low complexity probability-based piecewise linear approximation of the sigmoid function [J]. Journal of Xidian University, 2020, 47(3): 58-65.
[11]	WANG Jijun,HAO Ziyu,LI Hongliang. Optimization of memory access for the convolutional neural network training [J]. Journal of Xidian University, 2020, 47(2): 98-107.
[12]	LIU Lei,JIA Renxu. Analysis of sensitivity uncertainty of the MEMS microphone based on Latin hypercube Monte Carlo simulation [J]. Journal of Xidian University, 2019, 46(6): 23-29.
[13]	CAO Weidong,LI Jiaqi,WANG Huaichao. Analysis of targeted sentiment by the attention gated convolutional network model [J]. Journal of Xidian University, 2019, 46(6): 30-36.
[14]	LIU Mingqian,LI Kunming,WANG Chuanchuan,ZHANG Shun. Method for comprehensive evaluation of in-pulse characteristics of radar emitter signals [J]. Journal of Xidian University, 2019, 46(5): 8-14.
[15]	ZHU Suya,DU Jianchao,LI Yunsong,WANG Xiaopeng. Method for bridge crack detection based on the U-Net convolutional networks [J]. Journal of Xidian University, 2019, 46(4): 35-42.