维度情感模型下的表情图像生成及应用

doi:10.19665/j.issn1001-2400.2021.05.005

Abstract

Abstract:

In order to solve the problem that the training data of deep learning based facial expression recognition methods usually cover a limited part of the expression space and have an imbalanced distribution,we propose AV-GAN,a facial expression image synthesis method in Arousal-Valence dimensional emotion space,based on the generative adversarial network,to generate more diverse and balanced facial expression training data.The method uses label distribution to represent the expression for the face image,and employs an identity control module,an expression control module,and adversarial learning to realize the random sampling and generation of expression images in Arousal-Valence space.Evaluations on Oulu-CASIA database show that the accuracy of the recognition of the facial expression using the proposed method to augment training data is increased by 6.5%,compared with that using the original training data.It is proved that the proposed method can effectively improve the facial expression recognition accuracy under imbalanced training data.

Key words: Arousal-Valence emotion model, generative adversarial network, image generation, data augmentation, facial expression recognition

CLC Number:

TP391

YANG Jingbo,ZHAO Qijun,LYU Zejun. Synthesis of the expression image and its application under the dimentional emotion model[J].Journal of Xidian University, 2021, 48(5): 30-37.

Figures/Tables 10

References 20

[1]	LI S, DENG W H. Deep Facial Expression Recognition:A Survey[C]// IEEE Transactions on Affective Computing.Piscataway:IEEE, 2020:1-1.
[2]	EKMAN P, FRIESEN W V. Constants Across Cultures in the Face and Emotion[J]. Journal of Personality and Social Psychology, 1971, 17(2):124-129. doi: 10.1037/h0030377
[3]	杨勇, 黄文波, 金裕成, 等. 基于MSVR和Arousal-Valence情感模型的表情识别研究[J]. 重庆邮电大学学报(自然科学版), 2016, 28(6):836-843.
	YANG Yong, HUANG Wenbo, JIN Yucheng, et al. Facial Expression Recognition Method Based on MSVR and Arousal-Valence Emotion Model[J]. Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition), 2016, 28(6):836-843.
[4]	耿新, 徐宁. 标记分布学习与标记增强[J]. 中国科学:信息科学, 2018, 48(5):521-530.
	GENG Xin, XU Ning. Label Distribution Learning and Label Enhancement (in Chinese)[J]. Sci Sin Inform, 2018, 48(5):521-530.
[5]	MOLLAHOSSEINI A, HASSANI B, MAHOOR M H. AffectNet:A Database for Facial Expression,Valence,and Arousal Computing in the Wild[J]. IEEE Transactions on Affective Computing, 2019, 10(1):18-31. doi: 10.1109/TAFFC.2017.2740923
[6]	EKMAN, P, FRIESEN, W V. Facial Action Coding System (FACS):A Technique for the Measurement of Facial Actions[J]. Rivista Di Psichiatria, 1978, 47(2):126-38.
[7]	HE Z L, ZUO W M, KAN M, et al. AttGAN:Facial Attribute Editing by Only Changing What You Want[J]. IEEE Transactions on Image Processing, 2019, 28(11):5464-5478. doi: 10.1109/TIP.83
[8]	YANG H Y, ZHANG Z, YIN L J. Identity-Adaptive Facial Expression Recognition through Expression Regeneration Using Conditional Generative Adversarial Networks[C]// 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).Piscataway:IEEE, 2018:294-301.
[9]	WANG W X, SUN Q, FU Y W, et al. Comp-GAN:Compositional Generative Adversarial Network in Synthesizing and Recognizing Facial Expression[C]// Proceedings of the 27th ACM International Conference on Multimedia.Nice:ACM, 2019:211-219.
[10]	DING H, SRICHARAN K, CHELLAPPA R. ExprGAN:Facial Expression Editing with Controllable Expression Intensity[C]// The Thirty-Second AAAI Conference on Artificial Intelligence.Louisiana:AAAI, 2018:6781-6788.
[11]	LINDT A, BARROS P, SIQUEIRA H, et al. Facial Expression Editing with Continuous Emotion Labels[C]// 2019 14th IEEE International Conference on Automatic Face Gesture Recognition.Piscataway:IEEE, 2019:1-8.
[12]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: MIT Press, 2014:2627-2680.
[13]	莫建文, 徐凯亮, 林乐平, 等. 结合互信息最大化的文本到图像生成方法[J]. 西安电子科技大学学报, 2019, 46(5):180-188.
	MO Jianwen, XU Kailiang, LIN Leping, et al. Text-to-Image Generation Combined with Mutual Information Maximization[J]. Journal of Xidian Universyt, 2019, 46(5):180-188.
[14]	贾宇峰, 马力. 条件约束下的自我注意生成对抗网络[J]. 西安电子科技大学学报, 2019, 46(6):163-171.
	JIA Yufeng, MA Li. Self-Attention Generative Adversarial Network with the Conditional Constraint[J]. Journal of Xidian Universyt, 2019, 46(6):163-171.
[15]	PARKHI O M, VEDALDI A, ZISSERMAN A. Deep Face Recognition[C]//The British Machine Vision Conference (BMVC 2015).Swansea:BMVC, 2015: 41.1-41.12.
[16]	HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition[C]// The 29th IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE, 2016:770-778.
[17]	ZHAO G Y, HUANG X H, TAINI M, et al. Facial Expression Recognition from Near-Infrared Videos[J]. Image and Vision Computing, 2011, 29(9):607-619. doi: 10.1016/j.imavis.2011.07.002
[18]	ZHANG K P, ZHANG Z P, LI Z F, et al. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks[J]. IEEE Signal Processing Letters, 2016, 23(10):1499-1503. doi: 10.1109/LSP.2016.2603342
[19]	SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[J/OL]. [2020-07-10]. https://arxiv.org/abs/1409.1556.
[20]	KINGMA D P, BA J. Adam:A Method for Stochastic Optimization[C/OL]. [2020-07-10]. https://arxiv.org/abs/1412.6980.

表情类别	训练数据	测试数据
生气	1 267	443
厌恶	1 227	326
恐惧	1 357	359
高兴	1 368	343
悲伤	1 235	363
惊讶	1 259	362
中性	366	114
合计	8 079	2 300

表情类别	训练数据	测试数据
生气	2 500	443
厌恶	2 500	326
恐惧	2 500	359
高兴	2 500	343
悲伤	2 500	353
惊讶	2 500	362
中性	2 500	114
合计	2 500	2 300

表情类别	基准方法	传统方法(随机镜像)^[1]	传统方法(随机旋转)^[1]	传统方法(随机裁剪)^[1]	AV-GAN方法
生气	42.50±15.69	51.25±8.16	48.25±5.07	48.75±12.09	54.25±7.79
厌恶	41.00±4.30	38.75±5.21	42.00±7.65	47.50±9.09	56.00±10.56
恐惧	55.00±16.17	50.25±14.04	52.00±8.90	44.75±17.15	52.25±12.71
高兴	63.75±4.76	72.75±6.57	72.25±5.11	74.50±8.67	71.50±3.28
悲伤	51.75±6.87	51.25±12.34	46.00±7.58	51.50±7.53	49.25±8.04
惊讶	64.25±3.11	67.00±4.30	61.75±3.77	65.75±6.87	67.75±8.50
中性	3.50±2.29	2.00±1.58	1.00±0.00	3.75±1.09	29.50±7.43
合计	50.25±4.76	52.75±5.11	51.00±3.94	52.75±6.01	56.75±6.98

Synthesis of the expression image and its application under the dimentional emotion model

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 20

Related Articles 14

Metrics

Comments

Recommended 10

[1]	YU Haoyang,YIN Liang,LI Shufang,LV Shun. Recognition algorithm for the little sample radar modulation signal based on the generative adversarial network [J]. Journal of Xidian University, 2021, 48(6): 96-104.
[2]	SUN Yanjing,WEI Li,ZHANG Nianlong,YUN Xiao,DONG Kaiwen,GE Min,CHENG Xiaozhou,HOU Xiaofeng. Person re-identification method combining the DD-GAN and Global feature in a coal mine [J]. Journal of Xidian University, 2021, 48(5): 201-211.
[3]	WANG Junjun,SUN Yue,LI Ying. Cloud removal method for the remote sensing image based on the GAN [J]. Journal of Xidian University, 2021, 48(5): 23-29.
[4]	WEI Ziyu,YANG Xi,WANG Nannan,YANG Dong,GAO Xinbo. Reciprocal bi-directional generative adversarial network for cross-modal pedestrian re-identification [J]. Journal of Xidian University, 2021, 48(2): 205-212.
[5]	JIANG Shaobin,DU Chun,CHEN Hao,LI Jun,WU Jiangjiang. Unsupervised adversarial learning method for hard disk failure prediction [J]. Journal of Xidian University, 2020, 47(2): 118-125.
[6]	ZHANG Zhiyuan,DIAO Yinghua. Pedestrian trajectory prediction model with social features and attention [J]. Journal of Xidian University, 2020, 47(1): 10-17.
[7]	YANG Xiaoli,LIN Suzhen. Method for multi-band image feature-level fusion based on the attention mechanism [J]. Journal of Xidian University, 2020, 47(1): 120-127.
[8]	JIA Yufeng,MA Li. Self-attention generative adversarial network with the conditional constraint [J]. Journal of Xidian University, 2019, 46(6): 163-171.
[9]	SHEN Haijie,BIAN Qian,CHEN Xiaofan,WANG Zhenduo,TIAN Xinzhi. Video deblurring using the generative adversarial network [J]. Journal of Xidian University, 2019, 46(6): 112-117.
[10]	MO Jianwen,XU Kailiang,LIN Leping,OUYANG Ning. Text-to-image generation combined with mutual information maximization [J]. Journal of Xidian University, 2019, 46(5): 180-188.
[11]	CHEN Xiaofan,SHEN Haijie,BIAN Qian,WANG Zhenduo,TIAN Xinzhi. Face image super-resolution with an attention mechanism [J]. Journal of Xidian University, 2019, 46(3): 148-153.
[12]	XU Qiang;LI Wei;ZHAN Ronghui;ZOU Kun. Improved algorithm for SAR target recognition based on the convolutional neural network [J]. Journal of Xidian University, 2018, 45(5): 177-183.
[13]	ZHANG Jingang;FANG Yuan;YUAN Hao;WANG Shuzhen. Multiple convolutional neural networks for facial expression sequence recognition [J]. Journal of Xidian University, 2018, 45(1): 150-155.
[14]	FENG Xiaoyi;LAI Yangming;WANG Wenxing;CUI Shaoxing;PENG Jinye;JIANG Xiaoyu. Facial expression recognition based on the binary code of edges [J]. J4, 2015, 42(3): 186-191.