一种基于局部表征的面部表情识别算法

doi:10.19665/j.issn1001-2400.2021.05.013

Abstract

Abstract:

Expression is an important embodiment of human inner emotion change.Current expression recognition methods usually rely on global facial features,ignoring local features extraction.Psychologists point out that different facial expressions correspond to different regions of local muscle movement.In this paper,we propose an expression recognition algorithm based on local representation,referred to as EAU-CNN.In order to extract the local features of the face,the whole face image is first divided into 43 sub-regions according to the 68 feature points of the face.Then,8 local candidate regions covered by the muscle motion region and the facial salient organs are selected as the input of the convolution neural network.In order to balance the features of local candidate regions,the EAU-CNN adopts 8 parallel feature extraction branches,each of which dominates the full connected layer of different dimensions.The outputs of the branches are adaptively connected in terms of attention to highlight the importance of different local candidate regions.Finally,the expressions are divided into 7 categories:neutral,angry,disgusted,surprised,happy,sad and afraid by the Softmax function.In this paper,the algorithm is evaluated on CK +,JAFFE and custom FED datasets.The average accuracy of the proposed algorithm is 99.85%,96.61% and 98.6%,respectively.The evaluation index is 6.01%,10.17%,6.09% higher than that of the S-Patches algorithm.The results show that local representation can improve the performance of emotional recognition.

Key words: expression recognition, motion unit partition, convolutional neural network, loss function

CLC Number:

TP183

CHEN Changchuan,WANG Haining,HUANG Lian,HUANG Tao,LI Lianjie,HUANG Xiangkang,DAI Shaosheng. Facial expression recognition based on local representation[J].Journal of Xidian University, 2021, 48(5): 100-109.

Figures/Tables 18

References 22

[1]	徐峰, 张军平. 人脸微表情识别综述[J]. 自动化学报, 2017, 43(3):333-348.
	XU Feng, ZHANG Junping. Facial Expression Recognition:A Survey[J]. Acta Automatica Sinica, 2017, 43(3):333-348.
[2]	MEHRABIAN A. Communication without Words[J]. Psychology Today, 1968, 2(4):53-56.
[3]	EKMAN P, FRIESEN W V. Facial Action Coding System:A Technique for the Measurement of Facial Movement[M]. Palo Alto: Consulting Psychologists Press, 1978.
[4]	PANTIC M, PATRAS I. Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J]. IEEE Transactions on Systems,Man,and Cybernetics,Part B (Cybernetics), 2006, 36(2):433-449. doi: 10.1109/TSMCB.2005.859075
[5]	COHEN I, SEBE N, GARG A, et al. Facial Expression Recognition from Video Sequences:Temproal and Static Modeling[J]. Computer Vision and Image Understanding, 2003, 91(1-2):160-187. doi: 10.1016/S1077-3142(03)00081-X
[6]	邓捷方, 杨勇, 王国胤. 一种基于局部特征融合的表情识别方法[J]. 重庆邮电大学学报(自然科学版), 2011, 23(5):626-630.
	DENG Jiefang, YANG Yong, WANG Guoyin. An Expression Recognition Method Based on Local Feature Fusion[J]. Journal of Chongqing University of Posts and Telecommunications, 2011, 23(5):626-630.
[7]	杨勇, 蔡舒博. 一种基于两步降维和并行特征融合的表情识别方法[J]. 重庆邮电大学学报(自然科学版), 2015, 27(3):377-385.
	YANG Yong, CAI Shubo. Facial Expression Recognition Method Based on Two-Steps Dimensionality Reduction and Parallel Feature Fusion[J]. Journal of Chongqing University of Posts and Telecommunications, 2015, 27(3):377-385.
[8]	LAJEVARDI S M, HUSSAIN Z M. Automatic Facial Expression Recognition:Feature Extraction and Selection[J]. Signal,Image and Video Processing, 2012, 6:159-169. doi: 10.1007/s11760-010-0177-5
[9]	LUCEY P, COHN J F, KANADE T, et al. The Extended Cohn-Kanade Dataset(CK+):A Complete Dataset for Action Unit and Emotion-Specified Expression[C]// Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops.Piscataway:IEEE, 2010:94-101.
[10]	LYONS M, AKAMATSU S, KAMACHI M, et al. Coding Facial Expressions with Gabor Wavelets[C]// Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.Piscataway:IEEE, 1998:1-6.
[11]	YANG B, CAO J, NI R, et al. Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images[J]. IEEE Access, 2018, 6:4630-4640. doi: 10.1109/ACCESS.2017.2784096
[12]	MOLLAHOSSEINI A, CHAN D, MAHOOR M H, et al. Going Deeper in Facial Expression Recognition Using Deep Neural Networks[C]// 2016 IEEE Winter Conference on Applications of Computer Vision.Piscataway:IEEE, 2016:1-10.
[13]	KAHOUS E, PAL C, BOUTHILLIER X, et al. Combing Modality Specific Deep Neural Networks for Emotion Recognition in Video[C]// Proceedings of the 15th ACM on International Conference on Multimodal Interaction.New York:ACM, 2013:543-550.
[14]	HINTON G, DENG L, YU D, et al. Deep Neural Networks for Acoustic Modeling in Speech Recognition:The Shared Views of Four Research Groups[J]. IEEE Signal Processing Magazine, 2012, 29(6):82-97.
[15]	冯晓毅, 赖阳明, 王文星等. 一种边缘二进制码的表情识别方法[J]. 西安电子科技大学学报, 2015, 42(3):186-191.
	FENG Xiaoyi, LAI Yangming, WANG Wenxing, et al. Facial Expression Recognition Based on the Binary Code of Edges[J]. Journal of Xidian University, 2015, 42(3):186-191.
[16]	谭小慧, 李昭伟, 樊亚春. 基于多尺度细节增强的面部表情识别方法[J]. 电子与信息学报, 2019, 41(11):2752-2759.
	TAN Xiaohui, LI Zhaowei, FAN Yachun. Facial Expression Recognition Method Based on Multi-Scale Detail Enhancement[J]. Journal of Electronics & Information Technology, 2019, 41(11):2572-2759.
[17]	HAPPYS L, ROUTRAY A. Automatic Facial Expression Recognition Using Features of Salient Facial Patches[J]. IEEE Transactions on Affective Computing, 2015, 6(1):1-12. doi: 10.1109/TAFFC.2014.2386334
[18]	KALEEKAL T, SINGH J. Facial Expression Recognition Using Hahn Moment on Facial Patches[C]// 2019 4th International Conference on Recent Trends on Electronics,Information,Communication & Technology (RTEICT).Piscataway:IEEE, 2019:1338-1344.
[19]	LI Y, WANG S, ZHAO Y, et al. Simultaneous Facial Feature Tracking and Facial Expression Recognition[J]. IEEE Transactions on Image Processing, 2013, 22(7):2559-2573. doi: 10.1109/TIP.2013.2253477
[20]	LOPESA T, AGUIAR E D, SOUZA A F D, et al. Facial Expression Recognition with Convolutional Neural Networks:Coping with Few Data and The Training Sample Order[J]. Pattern Recognition, 2017, 61:610-628. doi: 10.1016/j.patcog.2016.07.026
[21]	李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别[J]. 自动化学报, 2018, 44(1):176-182.
	LI Yong, LIN Xiaozhu, JIANG Mengying. Facial Expression Recognition with Cross-Connect LeNet-5 Network[J]. Acta Automatica Sinica, 2018, 44(1):176-182.
[22]	宋剑桥, 王峰, 牛锦, 等. 一种面向时空神经网络的潜在情绪识别方法[J]. 西安电子科技大学学报, 2021, 48(4):159-167.
	SONG Jianqiao, WANG Feng, NIU Jin, et al. Potential Emotion Recognition Based on the Fusion of the Spatio-Temporal Neural Network and Facial Pulse Signals[J]. Journal of Xidian University, 2021, 48(4):159-167.

AU	描述	样图
1	内眉抬起
2	外眉抬起
4	眉毛压低
5	上眼睑抬起
6	脸颊抬起
7	眼皮拉紧
9	鼻子皱起
10	上嘴唇抬起
12	嘴角拉紧
15	嘴角下拉
17	下巴抬起
20	嘴唇拉长
23	嘴唇紧闭
24	嘴唇下拉
26	下颚下拉

表情	相应AUs
愤怒	AU4,AU5,AU7,AU23,AU24
厌恶	AU9,AU17
恐惧	AU4,AU1+AU5,AU5+AU7
高兴	AU6,AU12,AU25
悲伤	AU1,AU4,AU15,AU17
惊讶	AU5,AU26,AU27,AU1+AU2

AU局部候选区域	AU编号	特征区域	产生表情
AUg₁	AU1,AU2,AU5,AU7	1,2,5,6,8,9,12,13,40,41,42,43	愤怒、恐惧、悲伤、惊讶
AUg₂	AU4	1,2,3,4,5,6,8,9,12,13,40,41	愤怒、恐惧、悲伤
AUg₃	AU6	16,17,18,19,42,43	高兴
AUg₄	AU9	10,11,17,18,22,23	厌恶
AUg₅	AU12,AU15	21,22,23,24,25,26,27,28,37	高兴、悲伤、惊讶
AUg₆	AU25,AU26,AU27	25,26,27,28,29,30,31,32,33,34,35,36,37	高兴、惊讶
AUg₇	AU17	29,30,31,32,33,34,35,36	厌恶、悲伤
AUg₈	AU23,AU24	26,27,29,30,31,32,37	愤怒

AU组名	特征区域1	特征区域2
AUg₁	1,2,5,6,8,9,12,13,40,41	7,8,9,12,13,14,40,41,42,43
AUg₂	8,9,10,11,12,13,40,41,42,43	1,2,3,4,5,6,10,11
AUg₃	16,17,18,19	16,17,18,19,22,23
AUg₄	17,18,22,23	10,11,17,18,19,22,23
AUg₅	22,23,26,27,37	15,20,21,22,23,24
AUg₆	25,26,27,28,37	25,26,27,28,37
AUg₇	29,30,31,32,33,34,35	29,30,31,32,33,34,35,36
AUg₈	26,27,37	22,23,26,27,37

表情	中性	愤怒	厌恶	恐惧	高兴	悲伤	惊讶
中性	100.00	0	0	0	0	0	0
愤怒	0	100.00	0	0	0	0	0
厌恶	0	0	100.00	0	0	0	0
恐惧	0	0	0	100.00	0	0	0
高兴	0	0	0	0	100.00	0	0
悲伤	0	0	0	0	0	100.00	0
惊讶	0	0	0	0	0	0	100.00

Facial expression recognition based on local representation

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 18

References 22

Related Articles 15

Metrics

Comments

Recommended 10

算法	中性	愤怒	厌恶	恐惧	高兴	悲伤	惊讶	平均	CPU每帧耗时/ms
S-Patches^[16]	83.29	88.74	99.12	98.43	89.97	90.64	95.22	92.20	5.86
M-scale^[17]	90.80	90.37	92.45	97.37	95.63	92.10	97.54	93.75	6.43
CNN^[20]	98.79	92.15	94.28	97.12	100.00	98.54	93.21	96.30	1 351.01
STNN^[22]	79.24	85.12	90.87	89.20	70.56	78.34	80.51	81.98	1.23
EAU-CNN	98.78	98.37	98.74	97.69	100.00	97.47	100	98.29	50.25

[1]	YU Haoyang,YIN Liang,LI Shufang,LV Shun. Recognition algorithm for the little sample radar modulation signal based on the generative adversarial network [J]. Journal of Xidian University, 2021, 48(6): 96-104.
[2]	SUN Yanjing,WEI Li,ZHANG Nianlong,YUN Xiao,DONG Kaiwen,GE Min,CHENG Xiaozhou,HOU Xiaofeng. Person re-identification method combining the DD-GAN and Global feature in a coal mine [J]. Journal of Xidian University, 2021, 48(5): 201-211.
[3]	YANG Jingbo,ZHAO Qijun,LYU Zejun. Synthesis of the expression image and its application under the dimentional emotion model [J]. Journal of Xidian University, 2021, 48(5): 30-37.
[4]	ZHOU Peng,YANG Jun. Semantic segmentation of remote sensing images based on neural architecture search [J]. Journal of Xidian University, 2021, 48(5): 47-57.
[5]	YANG Yunhang,MIN Lianquan. Multi-scalefusion sketch recognition model by dilated convolution [J]. Journal of Xidian University, 2021, 48(5): 92-99.
[6]	SONG Jianfeng,MIAO Qiguang,WANG Chongxiao,XU Hao,YANG Jin. Multi-scale single object tracking based on the attention mechanism [J]. Journal of Xidian University, 2021, 48(5): 110-116.
[7]	ZHANG Yuhao,CHENG Peitao,ZHANG Shuhao,WANG Xiumei. Lightweight image super-resolution with the adaptive weight learning network [J]. Journal of Xidian University, 2021, 48(5): 15-22.
[8]	HUI Haisheng,ZHANG Xueying,WU Zelin,LI Fenglian. Method for stroke lesion segmentation using the primary-auxiliary path attention compensation network [J]. Journal of Xidian University, 2021, 48(4): 200-208.
[9]	WANG Ping,JIANG Yuze,ZHAO Guanghui. Object detection based on the multiscale location Enhancement network [J]. Journal of Xidian University, 2021, 48(3): 85-90.
[10]	GUO Zekun,TIAN Long,HAN Ning,WANG Penghui,LIU Hongwei,CHEN Bo. Radar HRRP based few-shot target recognition with CNN-SSD [J]. Journal of Xidian University, 2021, 48(2): 7-14.
[11]	CHENG Lei,WANG Yue,TIAN Chunna. Residual attention mechanism for visual tracking [J]. Journal of Xidian University, 2020, 47(6): 148-157.
[12]	KONG Xin,CHEN Gang,GONG Guoliang,LU Huaxiang,Mao Wenyu. High performance multiply-accumulator for the convolutional neural networks accelerator [J]. Journal of Xidian University, 2020, 47(4): 55-63.
[13]	DANG Jisheng,YANG Jun. 3D model recognition and segmentation based on multi-feature fusion [J]. Journal of Xidian University, 2020, 47(4): 149-157.
[14]	LI Kunlun,ZHANG Lu,XU Hongke,SONG Huansheng. Waveletdomain dilated network for fast low-dose CT image reconstruction [J]. Journal of Xidian University, 2020, 47(4): 86-93.
[15]	CUI Yanpeng,WANG Yuanhao,HU Jianwei. Detection method for a dynamic small target using the improved YOLOv3 [J]. Journal of Xidian University, 2020, 47(3): 1-7.