一种基于局部表征的面部表情识别算法

doi:10.19665/j.issn1001-2400.2021.05.013

摘要/Abstract

摘要：

表情是人类内心情感变化的重要体现。当前表情识别方法通常依赖面部全局特征进行处理,忽略局部特征提取。心理学家指出,不同面部表情对应不同的局部肌肉运动区域,以此为动机,提出一种基于局部表征的表情识别算法,简称EAU-CNN。为提取面部的局部特征,该文首先根据获取的人脸68个特征点将整体面部图像划分成43个子区域,随后选择肌肉运动区域与面部显著器官所覆盖的8个局部候选区域作为卷积神经网络的输入。为均衡局部候选区域的特征,EAU-CNN采用8个并行的特征提取分支,每个分支支配不同维全连接层。分支的输出按照注意力自适应地连接,以突出不同局部候选区域的重要程度。最后经Softmax函数,将表情分为中性、愤怒、厌恶、惊讶、高兴、悲伤和恐惧七类。该文在CK+、JAFFE、自定义大型FED数据集上对算法进行了评估实验,所提算法平均准确率分别为99.85%、96.61%、98.29%。该评价指标超过S-Patches算法6.01%、10.17%、6.09%,结果表明局部表征能够提升表情识别性能。

关键词: 表情识别, 运动单元分区, 卷积神经网络, 损失函数

Abstract:

Expression is an important embodiment of human inner emotion change.Current expression recognition methods usually rely on global facial features,ignoring local features extraction.Psychologists point out that different facial expressions correspond to different regions of local muscle movement.In this paper,we propose an expression recognition algorithm based on local representation,referred to as EAU-CNN.In order to extract the local features of the face,the whole face image is first divided into 43 sub-regions according to the 68 feature points of the face.Then,8 local candidate regions covered by the muscle motion region and the facial salient organs are selected as the input of the convolution neural network.In order to balance the features of local candidate regions,the EAU-CNN adopts 8 parallel feature extraction branches,each of which dominates the full connected layer of different dimensions.The outputs of the branches are adaptively connected in terms of attention to highlight the importance of different local candidate regions.Finally,the expressions are divided into 7 categories:neutral,angry,disgusted,surprised,happy,sad and afraid by the Softmax function.In this paper,the algorithm is evaluated on CK +,JAFFE and custom FED datasets.The average accuracy of the proposed algorithm is 99.85%,96.61% and 98.6%,respectively.The evaluation index is 6.01%,10.17%,6.09% higher than that of the S-Patches algorithm.The results show that local representation can improve the performance of emotional recognition.

Key words: expression recognition, motion unit partition, convolutional neural network, loss function

中图分类号:

TP183

陈昌川,王海宁,黄炼,黄涛,李连杰,黄向康,代少升. 一种基于局部表征的面部表情识别算法[J]. 西安电子科技大学学报, 2021, 48(5): 100-109.

CHEN Changchuan,WANG Haining,HUANG Lian,HUANG Tao,LI Lianjie,HUANG Xiangkang,DAI Shaosheng. Facial expression recognition based on local representation[J]. Journal of Xidian University, 2021, 48(5): 100-109.

图/表 18

表1

表2

图1

图2

表3

图3

图4

表4

图5

图6

图7

表5

图8

表6

图9

图10

图11

表7

参考文献 22

[1]	徐峰, 张军平. 人脸微表情识别综述[J]. 自动化学报, 2017, 43(3):333-348.
	XU Feng, ZHANG Junping. Facial Expression Recognition:A Survey[J]. Acta Automatica Sinica, 2017, 43(3):333-348.
[2]	MEHRABIAN A. Communication without Words[J]. Psychology Today, 1968, 2(4):53-56.
[3]	EKMAN P, FRIESEN W V. Facial Action Coding System:A Technique for the Measurement of Facial Movement[M]. Palo Alto: Consulting Psychologists Press, 1978.
[4]	PANTIC M, PATRAS I. Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J]. IEEE Transactions on Systems,Man,and Cybernetics,Part B (Cybernetics), 2006, 36(2):433-449. doi: 10.1109/TSMCB.2005.859075
[5]	COHEN I, SEBE N, GARG A, et al. Facial Expression Recognition from Video Sequences:Temproal and Static Modeling[J]. Computer Vision and Image Understanding, 2003, 91(1-2):160-187. doi: 10.1016/S1077-3142(03)00081-X
[6]	邓捷方, 杨勇, 王国胤. 一种基于局部特征融合的表情识别方法[J]. 重庆邮电大学学报(自然科学版), 2011, 23(5):626-630.
	DENG Jiefang, YANG Yong, WANG Guoyin. An Expression Recognition Method Based on Local Feature Fusion[J]. Journal of Chongqing University of Posts and Telecommunications, 2011, 23(5):626-630.
[7]	杨勇, 蔡舒博. 一种基于两步降维和并行特征融合的表情识别方法[J]. 重庆邮电大学学报(自然科学版), 2015, 27(3):377-385.
	YANG Yong, CAI Shubo. Facial Expression Recognition Method Based on Two-Steps Dimensionality Reduction and Parallel Feature Fusion[J]. Journal of Chongqing University of Posts and Telecommunications, 2015, 27(3):377-385.
[8]	LAJEVARDI S M, HUSSAIN Z M. Automatic Facial Expression Recognition:Feature Extraction and Selection[J]. Signal,Image and Video Processing, 2012, 6:159-169. doi: 10.1007/s11760-010-0177-5
[9]	LUCEY P, COHN J F, KANADE T, et al. The Extended Cohn-Kanade Dataset(CK+):A Complete Dataset for Action Unit and Emotion-Specified Expression[C]// Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops.Piscataway:IEEE, 2010:94-101.
[10]	LYONS M, AKAMATSU S, KAMACHI M, et al. Coding Facial Expressions with Gabor Wavelets[C]// Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 1998:1-6.
[11]	YANG B, CAO J, NI R, et al. Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images[J]. IEEE Access, 2018, 6:4630-4640. doi: 10.1109/ACCESS.2017.2784096
[12]	MOLLAHOSSEINI A, CHAN D, MAHOOR M H, et al. Going Deeper in Facial Expression Recognition Using Deep Neural Networks[C]// 2016 IEEE Winter Conference on Applications of Computer Vision.Piscataway:IEEE, 2016:1-10.
[13]	KAHOUS E, PAL C, BOUTHILLIER X, et al. Combing Modality Specific Deep Neural Networks for Emotion Recognition in Video[C]// Proceedings of the 15th ACM on International Conference on Multimodal Interaction.New York:ACM, 2013:543-550.
[14]	HINTON G, DENG L, YU D, et al. Deep Neural Networks for Acoustic Modeling in Speech Recognition:The Shared Views of Four Research Groups[J]. IEEE Signal Processing Magazine, 2012, 29(6):82-97.
[15]	冯晓毅, 赖阳明, 王文星等. 一种边缘二进制码的表情识别方法[J]. 西安电子科技大学学报, 2015, 42(3):186-191.
	FENG Xiaoyi, LAI Yangming, WANG Wenxing, et al. Facial Expression Recognition Based on the Binary Code of Edges[J]. Journal of Xidian University, 2015, 42(3):186-191.
[16]	谭小慧, 李昭伟, 樊亚春. 基于多尺度细节增强的面部表情识别方法[J]. 电子与信息学报, 2019, 41(11):2752-2759.
	TAN Xiaohui, LI Zhaowei, FAN Yachun. Facial Expression Recognition Method Based on Multi-Scale Detail Enhancement[J]. Journal of Electronics & Information Technology, 2019, 41(11):2572-2759.
[17]	HAPPYS L, ROUTRAY A. Automatic Facial Expression Recognition Using Features of Salient Facial Patches[J]. IEEE Transactions on Affective Computing, 2015, 6(1):1-12. doi: 10.1109/TAFFC.2014.2386334
[18]	KALEEKAL T, SINGH J. Facial Expression Recognition Using Hahn Moment on Facial Patches[C]// 2019 4th International Conference on Recent Trends on Electronics,Information,Communication & Technology (RTEICT).Piscataway:IEEE, 2019:1338-1344.
[19]	LI Y, WANG S, ZHAO Y, et al. Simultaneous Facial Feature Tracking and Facial Expression Recognition[J]. IEEE Transactions on Image Processing, 2013, 22(7):2559-2573. doi: 10.1109/TIP.2013.2253477
[20]	LOPESA T, AGUIAR E D, SOUZA A F D, et al. Facial Expression Recognition with Convolutional Neural Networks:Coping with Few Data and The Training Sample Order[J]. Pattern Recognition, 2017, 61:610-628. doi: 10.1016/j.patcog.2016.07.026
[21]	李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别[J]. 自动化学报, 2018, 44(1):176-182.
	LI Yong, LIN Xiaozhu, JIANG Mengying. Facial Expression Recognition with Cross-Connect LeNet-5 Network[J]. Acta Automatica Sinica, 2018, 44(1):176-182.
[22]	宋剑桥, 王峰, 牛锦, 等. 一种面向时空神经网络的潜在情绪识别方法[J]. 西安电子科技大学学报, 2021, 48(4):159-167.
	SONG Jianqiao, WANG Feng, NIU Jin, et al. Potential Emotion Recognition Based on the Fusion of the Spatio-Temporal Neural Network and Facial Pulse Signals[J]. Journal of Xidian University, 2021, 48(4):159-167.

AU	描述	样图
1	内眉抬起
2	外眉抬起
4	眉毛压低
5	上眼睑抬起
6	脸颊抬起
7	眼皮拉紧
9	鼻子皱起
10	上嘴唇抬起
12	嘴角拉紧
15	嘴角下拉
17	下巴抬起
20	嘴唇拉长
23	嘴唇紧闭
24	嘴唇下拉
26	下颚下拉

AU局部候选区域	AU编号	特征区域	产生表情
AUg₁	AU1,AU2,AU5,AU7	1,2,5,6,8,9,12,13,40,41,42,43	愤怒、恐惧、悲伤、惊讶
AUg₂	AU4	1,2,3,4,5,6,8,9,12,13,40,41	愤怒、恐惧、悲伤
AUg₃	AU6	16,17,18,19,42,43	高兴
AUg₄	AU9	10,11,17,18,22,23	厌恶
AUg₅	AU12,AU15	21,22,23,24,25,26,27,28,37	高兴、悲伤、惊讶
AUg₆	AU25,AU26,AU27	25,26,27,28,29,30,31,32,33,34,35,36,37	高兴、惊讶
AUg₇	AU17	29,30,31,32,33,34,35,36	厌恶、悲伤
AUg₈	AU23,AU24	26,27,29,30,31,32,37	愤怒

AU组名	特征区域1	特征区域2
AUg₁	1,2,5,6,8,9,12,13,40,41	7,8,9,12,13,14,40,41,42,43
AUg₂	8,9,10,11,12,13,40,41,42,43	1,2,3,4,5,6,10,11
AUg₃	16,17,18,19	16,17,18,19,22,23
AUg₄	17,18,22,23	10,11,17,18,19,22,23
AUg₅	22,23,26,27,37	15,20,21,22,23,24
AUg₆	25,26,27,28,37	25,26,27,28,37
AUg₇	29,30,31,32,33,34,35	29,30,31,32,33,34,35,36
AUg₈	26,27,37	22,23,26,27,37

算法	中性	愤怒	厌恶	恐惧	高兴	悲伤	惊讶	平均	CPU每帧耗时/ms
S-Patches^[16]	83.29	88.74	99.12	98.43	89.97	90.64	95.22	92.20	5.86
M-scale^[17]	90.80	90.37	92.45	97.37	95.63	92.10	97.54	93.75	6.43
CNN^[20]	98.79	92.15	94.28	97.12	100.00	98.54	93.21	96.30	1 351.01
STNN^[22]	79.24	85.12	90.87	89.20	70.56	78.34	80.51	81.98	1.23
EAU-CNN	98.78	98.37	98.74	97.69	100.00	97.47	100	98.29	50.25