Scientific Journal

Applied Aspects of Information Technology

The application of deep learning convolutional neural networks for solving the problem of automated facial expression  recognition and determination of emotions of a person is analyzed. It is proposed to use the advantages of the transfer approach to  deep learning convolutional neural networks training to solve the problem of insufficient data volume in sets of images with different  facial expressions. Most of these datasets are labeled in accordance with a facial coding system based on the units of human facial  movement. The developed technology of transfer learning of the public deep learning convolutional neural networks families  DenseNet and MobileNet, with the subsequent “fine tuning” of the network parameters, allowed to reduce the training time and  computational resources when solving the problem of facial expression recognition without losing the reliability of recognition of  motor units. During the development of deep learning technology for convolutional neural networks, the following tasks were solved.  Firstly, the choice of publicly available convolutional neural networks of the DenseNet and MobileNet families pre-trained on the  ImageNet dataset was substantiated, taking into account the peculiarities of transfer learning for the task of recognizing facial  expressions and determining emotions. Secondary, a model of a deep convolutional neural network and a method for its training have  been developed for solving problems of recognizing facial expressions and determining human emotions, taking into account the  specifics of the selected pretrained convolutional neural networks. Thirdly, the developed deep learning technology was tested, and  finally, the resource intensity and reliability of recognition of motor units on the DISFA set were assessed. The proposed technology  of deep learning of convolutional neural networks can be used in the development of systems for automatic recognition of facial  expressions and determination of human emotions for both stationary and mobile devices. Further modification of the systems for  recognizing motor units of human facial activity in order to increase the reliability of recognition is possible using of the augmentation technique.

1. Samadiani, N., Huang, G., Cai, B., Luo, W., Chi, C., Xiang, Y. & He, J. “A Review on Automatic  Facial Expression Recognition Systems Assisted by Multimodal Sensor Data”. Sensors. 2019; Vol. 19 No 8:  p. 1863. DOI: 

2. Li S. & Deng W. “Deep Facial Expression Recognition: A Survey”. IEEE Transactions on Affective  Computing. 2018. DOI:  

3. Le, Quoc Tuan , Antoshchuk, S. G., Nguyen, Thi Khanh Tien , Tran, The Vinh & Dang, Nhan Cach .  “Automated Student Attendance Monitoring System in Classroom Based on Convolutional Neural Net works”. Applied Aspects of Information Technology. Publ. Nauka i Tekhnika. Odessa: Ukraine. 2020; Vol. 3  No.3: 179–190. DOI: 

4. Nguyen, Thi Khanh Tien, Antoshchuk, S. G., Nikolenko, A. O., Tran, Kim Thanh & Babilunha, O. Yu. “Non-Stationary Time Series Prediction Using One-Dimensional Convolutional Neural Network Models”. Herald of Advanced Information Technology. Publ. Science i Technical. 2020; Vol.3 No.1: 362–372.  Odesa. Ukraine. DOI: 

5. “ImageNet: ImageNet overview”. – Available from: – [Accessed Dec2020]. 

6. Ekman, P. & Friesen, W. “Facial Action Coding System: A Technique for the Measurement of Facial  Movement”. Consulting Psychologists Press .1978. 

7. Ekman, P., Friesen, W.V. & Hager, J.C. “Facial Action Coding System (FACS)”. A Human Face. 2002. – Available from: dataface/facs/manual/TitlePage.html. 

8. Soysal, Ö. M., Shirzad, S. & K. Sekeroglu, K. “Facial Action Unit Recognition Using Data Mining  Integrated Deep Learning”. 2017 International Conference on Computational Science and Computational  Intelligence (CSCI), 2017. p. 437–443. DOI: 

9. Ekman, P. “Facial Expression and Emotion”. American Psychologist.1993; Vol. 48, No. 4: p.p.384- 392. 

10. Tian, Y.-I., Kanade, T. & Cohn, J. F. “Recognizing Action Units for Facial Expression Analysis”.  IEEE Transactions on Pattern Analysis and Machine Intelligence. 2001; Vol.23 No.2: 97–115.  DOI: 

11. Lin, Q. & He, R. & Jiang, P. “Feature Guided CNN for Baby’s Facial Expression Recognition”.  Complexity. 2020. p.1–10. DOI: 

12. Liu, M., Li, S., Shan, S., Wang, R. & Chen, X. “Deeply Learning Deformable Facial Action Parts  Model for Dynamic Expression Analysis”. Asian Conference on Computer VisionPubl. Springer. Berlin: Germany. 2014; Vol. 9006: 143–157. DOI: 

13. Jung, H., Lee, S., Yim, J., Park, S. & Kim, J. “Joint Fine-Tuning in Deep Neural Networks for Faci al Expression Recognition”. Proceedings of the IEEE International Conference on Computer Vision (ICCV). Santiago: Chile. 7–13 December, 2015. p. 2983–2991. DOI: 

14. Mollahosseini, A., Chan, D. & Mahoor, M.H. “Going deeper in facial expression recognition using  deep neural networks”. 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). 2016.  p.1–10. DOI: 

15. Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P. & Cohn, J. F.”DISFA: A Spontaneous Facial Action Intensity Database”. IEEE Transactions on Affective Computing. April-June, 2013; Vol. 4 No. 2:  151–160. DOI: 

16. Li, W., Abtahi, F., Zhu, Z., & Yin, L. “EAC-Net: Deep Nets with Enhancing and Cropping for Faci al Action Unit Detection”. In IEEE Transactions on Pattern Analysis and Machine Intelligence. Nov. 2018; Vol. 40 No. 11: 2583–2596. DOI: 

17. Zhang, Z., Zhai, S., & Yin, L. “Identity-based Adversarial Training of Deep CNNs for Facial Action  Unit Recognition”. British Machine Vision Conference. BMVA Press. 2018. 226 p. – Available from:  

18. Simonyan, K. & Zisserman. “A.Very Deep Convolutional Networks for Large-Scale Image Recog nition”. In: Bengio, Y., LeCun, Y. (eds.). 3rd International Conference on Learning Representations, ICLR San Diego: CA. USA. Conference Track Proceedings (May 7-9, 2015). – Available from: . 

19. Shao, Z., Liu, Z., Cai, J. & Ma, L. “JÂA-Net: Joint Facial Action Unit Detection and Face Align ment Via Adaptive Attention”. Int. J. Comput. Vis. 2021; Vol.129 No.2: 321–340. DOI: 10.1007/s11263-020-01378-z. 

20. Walecki, R., Rudovic, O., Pavlovic, V., Schuller, B. & Pantic, M. “Deep Structured Learning for  Facial Action Unit Intensity Estimation”. In: IEEE Conference on Computer Vision and Pattern Recognition  (CVPR). 2017. p. 5709–5718. DOI: 

21. Almaev, T., Martinez, B. & Valstar, M. “Learning to Transfer: Transferring Latent Task Structures  and Its Application to Person-Specific Facial Action Unit Detection”. In: IEEE International Conference on  Computer Vision. 2015. p. 3774–3782. DOI: 10.1109/ICCV.2015.430. 

22. Lim, Y., Liao, Z., Petridis, S. & Pantic, M. “Transfer Learning for Action Unit Recognition”. ArXiv. 2018. – Available from: .

23. Ntinou, I., Sanchez, E., Bulat, A., Valstar, M. & Tzimiropoulos, G. “A Transfer Learning Approach  to Heatmap Regression for Action Unit Intensity Estimation”. ArXiv. 2020. – Available from: 

24. Akhand, M. A. H., Roy, S., Siddique, N., Kamal, Md. A. S. & Shimamura, T. “Facial Emotion  Recognition Using Transfer Learning in the Deep CNN”. Electronics. 2021; Vol. 10 No 9.  DOI: electronics10091036. 

25. Bianco, S., Cadene, R., Celona, L. & Napoletano, P. “Benchmark Analysis of Representative Deep  Neural Network Architectures”. IEEE Access. 2018; Vol. 6: 64270–64277. DOI: ACCESS.2018.2877890. 

26. Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. “Densely Connected Convolutional Net works”. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017. p. 2261–2269. DOI: 

27. Howard, A., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M. & Adam,  H. “Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications”. ArXiv. 2017. – Available from: 

28. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L.C. “Mobilenetv2: Inverted Residuals  and Linear Bottlenecks”. IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018. p.4510–4520. DOI: 

29. Shao, Z., Liu, Z., Cai, J., Wu, Y. & Ma, L. “Facial Action Unit Detection Using Attention and Rela tion Learning”. IEEE Transactions on Active Computing. 2019. DOI: ta_c.2019.2948635 

30. Ioffe, S. & Szegedy, C. “Batch Normalization: Accelerating Deep Network Training by Reducing  Internal Covariate Shift”. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on  Machine Learning. Proceedings of Machine Learning Research. France. 07–09 Jul, 2015; Vol. 37: 448–456.  – Available from: 

31. Li, Y., Song, Y. & Luo, J. “Improving Pairwise Ranking for Multi-Label Image Classification”. In:  IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017. p. 1837–1845.  DOI: 

32. Arsirii O., Antoshchuk S., Babilunha O., Manikaeva O. & Nikolenko A. “Intellectual Information  Technology of Analysis of Weakly-Structured Multi-Dimensional Data of Sociological Research Lecture  Notes in Computational Intelligence and Decision Making”. Advances in Intelligent Systems and Computing. Publ. Springer, Cham. 2020; Vol. 1020: 242–258. DOI:
Last download:
22 Oct 2021


[ © KarelWintersky ] [ All articles ] [ All authors ]
[ © Odessa National Polytechnic University, 2018.]