双通道决策信息融合下的微表情识别

doi:10.19665/j.issn1001-2400.2022.04.015

摘要/Abstract

摘要：

微表情作为揭示潜在情绪的一个重要通道,是一种无意识的、不受大脑控制的非语言面部信息,能够反映人们最真实的心理感受和心理状态。但是,微表情存在动作幅度小且快速出现、不易被捕捉等特性,使得单一模态的微表情识别准确率难以提升。针对上述问题,提出一种微表情面部颜色特征提取算法,并将其提取的特征与微表情纹理特征进行决策融合,从而构成微表情双模态情感识别模型。该模型首先通过均匀动态纹理识别方法从经过预处理的微表情数据中提取相应的纹理特征;其次计算微表情两帧序列图片的每一个像素点之间的Lab色差,由此获得面部的颜色特征,并对其进行嵌入式特征选择以剔除冗余的特征;然后分别训练两种模态的分类器,并将两种模态训练得到的分类信息进行决策融合;最后得到微表情情绪分类结果。模型在微表情数据集CAMSE Ⅱ和SMIC上进行了实验。实验结果表明,微表情的纹理单模态和面部颜色单模态的平均识别准确率约为64.73%、51.64%和63.58%、50.48%,而决策融合后微表情情绪的识别结果约为68.11%和66.43%,高于融合前微表情的识别准确率,说明文中提出的微表情双模态情感识别模型明显提高了微表情的识别能力。

关键词: 面部颜色特征, 纹理特征, 欧拉视频放大, 特征选择, 决策融合

Abstract:

Micro-expression,as an important channel to reveal underlying emotion,is a kind of unconscious nonverbal facial information,not controlled by the brain,can reflect people's real psychological experience and psychological state,but micro-expression movements appear small and quick,and cannot be easily captured,making it difficult for a single mode of micro expression recognition accuracy to ascend.To solve the above problems,this paper proposes an algorithm for extracting the facial color features of micro-expressions,and integrates the extracted features with the texture features of micro-expressions for decision fusion,so as to construct the bi-modal emotion recognition model of micro-expressions.First,the model extracts the corresponding texture features from the preprocessed micro-expression data by the uniform LBP-TOP algorithm.Second,the Lab color difference between each pixel of two frames of micro-expression sequence images is calculated to obtain the facial color features,and the embedded feature selection is carried out to eliminate redundant features.Then,the classifiers of the two modes are trained respectively,and the classification information obtained from the two modes is fused for decision making.Finally,the classification results of micro-expressions are obtained.The model was tested on CAMSE Ⅱ and SMIC micro-expression dataset.Experimental results show that the average recognition accuracies of the micro-expression single mode of texture and face color are 64.73% and 51.64%,and 63.58% and 50.48%,while the results of micro-expression emotion recognition after decision fusion are 68.11% and 66.43%.The recognition accuracy is higher than that before the fusion,which indicates that the proposed bimodal emotion recognition model can significantly improve the recognition ability of micro expressions.

Key words: facial color feature, texture features, euler video amplification, feature selection, decision fusion

中图分类号:

TP183

戎如意,薛珮芸,白静,贾海蓉,谢娅利. 双通道决策信息融合下的微表情识别[J]. 西安电子科技大学学报, 2022, 49(4): 127-133.

RONG Ruyi,XUE Peiyun,BAI Jing,JIA Hairong,XIE Yali. Micro-expression recognition based on two-channel decision information fusion[J]. Journal of Xidian University, 2022, 49(4): 127-133.

图/表 7

图1

图2

图3

图4

表1

表2

表3

参考文献 24

[1]	贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述[J]. 计算机辅助设计与图形学学报, 2014, 26(9):1385-1395.
	BEN Xianye, YANG Mingqiang, ZHANG Peng, et al. Survey on Automatic Micro Expression Recognition Methods[J]. Journal of Computer-Aided Design & Computer Graphics, 2014, 26(9):1385-1395.
[2]	SEE J, YAP M H, LI J, et al. MEGC 2019--The Second Facial Micro-Expressions Grand Challenge[C]// 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).Piscataway:IEEE, 2019:1-5.
[3]	LIU Y, ZHANG J, YAN W, et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310. doi: 10.1109/TAFFC.2015.2485205
[4]	QI W, SHEN X, FU X. The Machine Knows What You Are Hiding:An Automatic Micro-Expression Recognition System[C]// Affective Computing and Intelligent Interaction -Fourth International Conference.Heidelberg:Springer, 2011:152-162.
[5]	XU F, ZHANG J, WANG J Z. Microexpression Identification and Categorization Using a Facial Dynamics Map[J]. IEEE Transactions on Affective Computing, 2017, 8(2):254-267. doi: 10.1109/TAFFC.2016.2518162
[6]	POLIKOVSKY S, KAMEDA Y, OHTA Y. Facial Micro-Expression Detection in Hi-Speed Video Based on Facial Action Coding System (FACS)[J]. IEICE Transactions on Information and Systems, 2013, 96(1):81-92.
[7]	ZHAO G, PIETIKÄINEN M. Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):915-928. doi: 10.1109/TPAMI.2007.1110
[8]	宋剑桥, 王峰, 牛锦, 等. 一种面向时空神经网络的潜在情绪识别方法[J]. 西安电子科技大学学报, 2021, 48(4):159-167.
	SONG Jianqiao, WANG Feng, NIU Jin, et al. Potential Emotion Recognition Based on the Fusion of the Spatio-Temporal Neural Network and Facial Pulse Signals[J]. Journal of Xidian University, 2021, 48(4):159-167.
[9]	王甦菁, 邹博超, 刘瑞, 等. 隐藏情绪分析与识别方法[J]. 心理科学进展, 2020, 28(9):1426-1436. doi: 10.3724/SP.J.1042.2020.01426
	WANG Sujing, ZOU Bochao, LIU Rui, et al. Concealed Emotion Analysis and Recognition Method[J]. Advances in Psychological Science, 2020, 28(9):1426-1436. doi: 10.3724/SP.J.1042.2020.01426
[10]	牛锦. 基于视频的潜在情绪识别研究[D]. 山西: 太原理工大学, 2020.
[11]	MARTINEZ A, BENITEZ-QUIROZ C, SRINIVASAN R. Facial Color Is an Efficient Mechanism to Visually Transmit Emotion[J]. Journal of Vision, 2018, 18(10):192. doi: 10.1167/18.10.192
[12]	冉杨鋆. 基于Lasso极限最小学习机和LAB颜色空间的人脸识别研究[D]. 上海: 复旦大学, 2012.
[13]	WU H, RUBINSTEIN M, SHIH E, et al. Eulerian Video Magnification for Revealing Subtle Changes in the World[J]. ACM Transactions on Graphics(TOG), 2012, 31(4):1-7.
[14]	刘汝涵, 徐丹. 视频放大和深度学习在微表情识别任务上的应用[J]. 计算机辅助设计与图形学学报, 2019, 31(9):1535-1541.
	LIU Ruhan, XU Dan. Video Amplification and Deep Learning in Micro-Expression Recognition[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(9):1535-1541.
[15]	ZHOU Z, ZHAO G, PIETIKÄINEN M. Towards a Practical Lipreading System[C]// The 24th IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2011:137-144.
[16]	YAN W J, LI X, WANGS J, et al. CASME II:An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation[J]. PlOS One, 2014, 9(1):1-8.
[17]	LI X, PFISTER T, HUANG X, et al. A Spontaneous Micro-Expression Database:Inducement,Collection and Baseline[C]// 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).Piscataway:IEEE, 2013:1-6.
[18]	PFISTE R T, LI X, ZHAO G, et al. Recognising Spontaneous Facial Micro-Expressions[C]// 2011 International Conference on Computer Vision.Piscataway:IEEE, 2012:1449-1456.
[19]	HUANG X, WANG S, ZHAO G, et al. Facial Micro Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection[C]// Proceedings of the IEEE International Conference on Computer Vision Workshop.Piscataway:IEEE, 2015:1-9.
[20]	LI J, WANG Y D,SEEJ, et al. Micro-Expression Recognition Based on 3D Flow Convolutional Neural Network[J]. Pattern Analysis and Applications, 2019, 22(4):1331-1339. doi: 10.1007/s10044-018-0757-5
[21]	KIM D, BADDAR W, RO Y. Micro-Expression Recognition with Expression-State Const-rained Spatio-Temporal Feature Representations (2016)[C/OL]. [2016-10-19]. https://dl.acm.org/doi/epdf/10.1145/2964284.2967247.
[22]	LIONG S, SEE J, WONG K, et al. Less is More:Micro-Expression Recognition from Video Using Apex Frame[J]. Signal Processing:Image Communication, 2018, 62:82-92. doi: 10.1016/j.image.2017.11.006
[23]	WU J, XU J, LIN D, et al. Optical Flow Filtering-Based Micro-Expression Recognition Method[J]. Electronics, 2020, 9(12):2056. doi: 10.3390/electronics9122056
[24]	梁正友, 刘德志, 孙宇. 结合迁移学习与可分离三维卷积的微表情识别方法[J]. 计算机工程, 2022, 48(1):228-235.
	LIANG Zhengyou, LIU Dezhi, SUN Yu. Micro-Expression Recognition Combining Transfer Learning and Separable 3D Convolution[J]. Computer Engineering, 2022, 48(1):228-235.

方法	CASME Ⅱ/%	SMIC(HS)/%
LBP-TOP	62.87	61.43
LBP-TOP+EVMa	64.73	63.58
色差计算	48.37	48.11
色差计算+EVMc	51.64	50.48
D-S决策融合	67.07	66.14
D-S决策融合+特征选择	68.11	66.43

方法	CASME Ⅱ/%	SMIC/%
LBP-TOP^[18]	63.41	62.80
STLBP-IP^[19]	59.93	59.51
3D-FCN^[20]	59.11	55.49
CNN-LSTM^[21]	60.98
Bi-WOOF^[22]	61.00	62.00
Optical Flow Filtering^[23]	61.57	65.41
S3DCNN-transfer^[24]	67.58
文中方法	68.11	66.43