Automatic Micro-Expression Recognition Using LBP-SIPl and FR-CNN

Document Type : Research Article

Authors

1 Faculty of Electrical and Computer Engineering, University of Tabriz, Tabriz, Iran

2 Department of Electrical Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran

Abstract

Facial Expressions are one of the most effective ways for non-verbal communications, which can be expressed as the Micro-Expression (ME) in the high-stake situations. The MEs are involuntary, rapid, subtle, and can reveal real human intentions. However, their feature extraction is very challenging due to their low intensity and very short duration. Although Local Binary Pattern on Three Orthogonal Plane (LBP-TOP) feature extractor is useful for ME analysis, it does not consider essential information. To address this problem, we propose a new feature extractor called Local Binary Pattern from Six Intersection Planes (LBP-SIPl). This method extracts LBP code on Six Intersection Planes, and then combines them. Results show that the proposed method has superior performance in apex frame spotting automatically in comparison with the relevant methods on the CASME I and the CASME II databases. Afterwards, the apex frames are the input of the Fast Region-based Convolutional Neural Network (FR-CNN) to recognize the Facial Expressions. Simulation results show that the ME has been automatically recognized in 81.56% and 96.11% on the CASME I and the CASME II databases by using the proposed method, respectively.

Keywords

Main Subjects

References

[1] S.-T. Liong, J. See, K. Wong, R. C.-W. Phan, Less is more: Micro-Expression recognition from video using apex frame, Signal Processing: Image Communication, 62 (2018) 82-92.
[2] X. Huang, S.-J. Wang, X. Liu, G. Zhao, X. Feng, M. Pietikäinen, Discriminative spatiotemporal Local Binary Pattern with revisited integral projection for spontaneous facial Micro-Expression recognition, IEEE Transactions on Affective Computing, 10(1) (2017) 32-47.
[3] X. Li et al., Towards reading hidden emotions: A comparative study of spontaneous Micro-Expression spotting and recognition methods, IEEE transactions on affective computing, 9(4) (2017) 563-577.
[4] V. Esmaeili, M. Mohassel Feghhi, S. O. Shahdi, Autonomous Apex Detection and Micro-Expression Recognition using Proposed Diagonal Planes, International Journal of Nonlinear Analysis and Applications, 11 (2020) 483-497.
[5] V. Esmaeili, M. M. Feghhi, S. O. Shahdi, Micro-Expression Recognition Using Histogram of Image Gradient Orientation on Diagonal Planes, in 2021 5th International Conference on Pattern Recognition and Image Analysis (IPRIA), (2021) 1-5: IEEE.
[6] W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, X. Fu, CASME database: A dataset of spontaneous Micro-Expressions collected from neutralized faces, in 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), (2013) 1-7: IEEE.
[7] W. Merghani, A. K. Davison, M. H. Yap, A review on facial Micro-Expressions analysis: datasets, features and metrics, arXiv preprint arXiv:1805.02397, (2018).
[8] X. Huang, G. Zhao, X. Hong, W. Zheng, M. Pietikäinen, Spontaneous facial Micro-Expression analysis using spatiotemporal completed local quantized patterns, Neurocomputing, 175 (2016) 564-578.
[9] E. A. Haggard, K. S. Isaacs, Micromomentary Facial Expressions as indicators of ego mechanisms in psychotherapy, in Methods of research in psychotherapy: Springer, (1966) 154-165.
[10] P. Ekman, W. V.  Friesen, Nonverbal leakage and clues to deception, Psychiatry, 32(1) (1969) 88-106.
[11] F. Xu, J. Zhang, J. Z. Wang, Microexpression identification and categorization using a facial dynamics map, IEEE Transactions on Affective Computing, 8(2) (2017) 254-267.
[12] J. A. Ruiz-Hernandez, M. Pietikäinen, Encoding Local Binary Patterns using the re-parametrization of the second order gaussian jet, in 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), (2013) 1-6: IEEE.
[13] B. Song et al., Recognizing spontaneous Micro-Expression using a three-stream Convolutional Neural Network, IEEE Access, 7 (2019) 184537-184551.
[14] S.-T. Liong, Y. S. Gan, J. See, H.-Q. Khor, Y.-C. Huang, Shallow Triple Stream Three-dimensional CNN (ststnet) for Micro-Expression recognition, in 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), (2019) 1-5: IEEE.
[15] Q. Li, J. Yu, T. Kurihara, H. Zhang, S. Zhan, Deep Convolutional Neural Network with Optical Flow for facial Micro-Expression recognition, Journal of Circuits, Systems and Computers, 29(01) (2020) 2050006,.
[16] L. Wang, J. Jia, N. Mao, Micro-Expression Recognition Based on 2D-3D CNN, in 2020 39th Chinese Control Conference (CCC), (2020) 3152-3157: IEEE.
[17] C. Wu, F. Guo, TSNN: Three‐Stream Combining 2D and 3D Convolutional Neural Network for Micro‐Expression Recognition, IEEJ Transactions on Electrical and Electronic Engineering, 16(1) (2021) 98-107.
[18] G. Zhao, M. Pietikainen, Dynamic texture recognition using Local Binary Patterns with an application to Facial Expressions, IEEE transactions on pattern analysis and machine intelligence, 29(6) (2007) 915-928.
[19] Y. Wang, J. See, R. C.-W. Phan, Y.-H. Oh, Lbp with six intersection points: Reducing redundant information in lbp-top for Micro-Expression recognition, in Asian conference on computer vision, (2014) pp. 525-537: Springer.
[20] V. Esmaeili, S. O. Shahdi, Automatic Micro-Expression apex spotting using Cubic-LBP, Multimedia Tools and Applications, 79(27) (2020) 20221-20239.
[21] V. Esmaeili, M. M. Feghhi, S. O. Shahdi, Automatic Micro-Expression Apex Frame Spotting using Local Binary Pattern from Six Intersection Planes, in Proc. of the 2020 International Conference on Machine Vision and Image Processing (MVIP), University of Tahran, Iran, Feb. 2020, available on: https://arxiv.org/pdf/2104.02149, (2021).
[22] T. Pfister, X. Li, G. Zhao, M. Pietikäinen, Recognising spontaneous facial Micro-Expressions, in 2011 international conference on computer vision, (2011) 1449-1456: IEEE.
[23] H. Ma, G. An, S. Wu, F. Yang, A Region Histogram of Oriented Optical Flow (RHOOF) feature for apex frame spotting in Micro-Expression, in 2017 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), (2017) 281-286: IEEE.
[24] R. Girshick, Fast r-cnn, in Proceedings of the IEEE international conference on computer vision, (2015) 1440-1448.
[25] S. O. Shahdi, S. A. R. Abu-Bakar, Neural network-based approach for face recognition across varying pose, International Journal of Pattern Recognition and Artificial Intelligence, 29(08) (2015) 1556015.
[26] T. Ojala, M. Pietikäinen, D. Harwood, A comparative study of texture measures with classification based on featured distributions, Pattern recognition, 29(1) (1996) 51-59.
[27] W.-J. Yan et al., CASME II: An improved spontaneous Micro-Expression database and the baseline evaluation, PloS one, 9(1) (2014) e86041.
[28] V. Esmaeili, M. Mohassel Feghhi, S. O. Shahdi, A comprehensive survey on facial Micro-Expression: approaches and databases, Multimedia Tools and Applications, (2022) 1-46.‏
[29] C. Nicholson, Evaluation metrics for machine learning—accuracy, precision, recall, and F1 defined, ed: Pathmind. http://pathmind.com/wiki/accuracy-precision-recall-f1, (2019).
[30] L. Zhou, Q. Mao, X. Huang, F. Zhang, Z. Zhang, Feature refinement: An expression-specific feature learning and fusion method for Micro-Expression recognition, Pattern Recognition, 122 (2022) 108275.
AUT Journal of Modeling and Simulation
Volume 54, Issue 1
June 2022
Pages 59-72

Files

Share

How to cite

Statistics