Histogram Based Shape and Textural Characteristics for Facial Emotion Recognition

Document Type : Research Article

Author

Department of Electrical Engineering (Communication), Tarbiat Modares University

Abstract

Emotion recognition has many applications in relation between human and machine. A facial emotion recognition framework for 6 basic emotions of happiness, sadness, disgust, surprise, anger and fear is proposed in this paper. The proposed framework utilizes the histogram estimate of shape and textural characteristics of face image. Instead of direct processing on the original gray levels of face image which may have not significant information about facial expression, the processing is done on transformed images containing informative features. The shape features are extracted by morphological operators by reconstruction and the texture ones are acquired by computing the gray-level co-occurrence matrix (GLCM), and applying Gabor filters. The use of whole face image may provide non-informative and redundant information. So, the proposed emotion recognition method just uses the most important components of face such as eyes, nose and mouth. After textural and shape feature extraction, the histogram function is applied to the shape and texture features containing emotional states of face. The simple and powerful nearest neighbor classifier is used for classification of fused histogram features. The experiments show the good performance of the proposed framework compared to some state-of-the-art facial expression methods such as local linear embedding (LLE), Isomap, Morphmap and local directional pattern (LDP).

Keywords

Main Subjects

References

         [1]         U. Thonse, R. V. Behere, S. K. Praharaj, P. S. V. N. Sharma, Facial emotion recognition, socio-occupational functioning and expressed emotions in schizophrenia versus bipolar disorder, Psychiatry Research, Available online 27 March 2018.
         [2]         D. K. Jain, P. Shamsolmoali, P. Sehdev, Extended deep neural network for facial emotion recognition, Pattern Recognition Letters, 120 (2019) 69-74.
         [3]         M. Z. Uddin, M. M. Hassan, A. Almogren, M. Zuair, G. Fortino, J. Torresen, A facial expression recognition system using robust face features from depth videos and deep learning, Computers & Electrical Engineering, 63 (2017) 114-125.
         [4]         D. Torres-Boza, M. C. Oveneke, F. Wang, D. Jiang, W. Verhelst, H. Sahli, Hierarchical sparse coding framework for speech emotion recognition, Speech Communication, 99 (2018) 80-89.
         [5]         A. Uribe, A. Gómez, M. Bastidas, O. L. Quintero, D. Campo, "A novel emotion recognition technique from voiced-speech," 2017 IEEE 3rd Colombian Conference on Automatic Control (CCAC), Cartagena (2017) 1-4.
         [6]         K. N. Minhad, S. H. M. Ali, M. B. I. Reaz, Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness, Journal of Transport & Health, 7 (Part A) (2017) 75-89.
         [7]         S. Vaid, P. Singh, "De-noising of EEG signal for emotion recognition," 2015 International Conference on Signal Processing, Computing and Control (ISPCC), Waknaghat (2015) 159-162.
         [8]         M. Z. Uddin, M. M. Hassan, A. Almogren, A. Alamri, M. Alrubaian, G. Fortino, "Facial Expression Recognition Utilizing Local Direction-Based Robust Features and Deep Belief Network," IEEE Access, 5 (2017) 4525-4536.
         [9]         G. Caridakis, K. Karpouzis, S. Kollias, User and context adaptive neural networks for emotion recognition, Neurocomputing 71 (2008) 2553– 2562.
        [10]        X. Huang, J. Kortelainen, G. Zhao, X. Li, A. Moilanen, T. Seppänen, M. Pietikäinen, Multi-modal emotion analysis from facial expressions and electroencephalogram, Computer Vision and Image Understanding, 147 (2016) 114-124.
        [11]        M. A. Nicolaou, H. Gunes, M. Pantic, Output-associative RVM regression for dimensional and continuous emotion prediction, Image and Vision Computing, 30 (3) (2012) 186-196.
        [12]        M. Song, M. You, N., C. Li, Chen, A robust multimodal approach for emotion recognition. Neurocomputing, 71 (2008) 1913-1920.
        [13]        P. Ekman, W.V. Friesen, Facial action coding system: A technique for the measurement of facial movement, palo alto. Consulting Psychologists Press, CA (1988); P.C. Ellsworth, C.A. Smith, From appraisal to emotion: Differences among unpleasant feelings, Motivation and Emotion 12 (1978) 271–302.
        [14]        Z. Li, J.i. Imai, M. Kaneko, Facial-component-based bag of words and phog descriptor for facial expression recognition, IEEE International Conference on Systems, Man and Cybernetics (SMC 2009) (2009) 1353–1358.
        [15]        M. Turk, A. Pentland, Face recognition using eigen faces, IEEE Conference on Computer Vision and Pattern Recognition, Maui, HI, USA (1991).
        [16]        T. Ahonen, A. Hadid, M. Pietikainen, Face description with local binary patterns: Application to face recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28 (2006) 2037–2041.
        [17]        J.B. Tenenbaum, V. de Silva, J.C. Langford, A global geometric framework for nonlinear dimensionality reduction, Science, 290 (5500) (2000) 2319–2323.
        [18]        S.T. Roweis, L.K. Saul, Nonlinear dimensionality reduction by locally linear embedding’, Science, 290 (5500) (2000) 2323–2326.
        [19]        R. Kapoor, R. Gupta, Morphological mapping for non-linear dimensionality reduction, IET Computer Vision, 9 (2) (2015) 226-233.
        [20]        T. Jabid, M. H. Kabir, O. Chae, Robust facial expression recognition based on local directional pattern, ETRI journal, 32 (5) (2010) 784–794.
        [21]        Y. Yoshitomi, T. Asada, R. Kato, M. Tabuse, Facial expression recognition using facial expression intensity characteristics of thermal image, J. Robot. Netw. Artif. Life, 2 (1) (2015) 5–8.
        [22]        S. Wang, B. Pan, H. Chen, Q. Ji, Thermal Augmented Expression Recognition, IEEE Transactions on Cybernetics, 48 (7) (2018) 2203-2214.
        [23]        M. Liu, S. Shan, R. Wang, X. Chen, Learning Expressionlets via Universal Manifold Model for Dynamic Facial Expression Recognition, IEEE Transactions on Image Processing, 25 (12) (2016) 5920-5932.
        [24]        Q. Mao, Q. Rao, Y. Yu, M. Dong, Hierarchical Bayesian Theme Models for Multipose Facial Expression Recognition, IEEE Transactions on Multimedia, 19 (4) (2017) 861-873.
        [25]        A. Majumder, L. Behera, V. K. Subramanian, Automatic Facial Expression Recognition System Using Deep Network-Based Data Fusion, IEEE Transactions on Cybernetics, 48 (1) (2018) 103-114.
        [26]        M. Z. Uddin, W. Khaksar, J. Torresen, Facial Expression Recognition Using Salient Features and Convolutional Neural Network, IEEE Access, 5 (2017) 26146-26161.
        [27]        D. K. Jain, Z. Zhang, K. Huang, Multi angle optimal pattern-based deep learning for automatic facial expression recognition, Pattern Recognition Letters, In Press.
        [28]        S. Wu, J. Xu, S. Zhu, H. Guo, A Deep Residual convolutional neural network for facial keypoint detection with missing labels, Signal Processing,Volume 144, 2018, Pages 384-391.
        [29]        L. Chen, M. Wu, M. Zhou, Z. Liu, J. She and K. Hirota, Dynamic Emotion Understanding in Human-Robot Interaction Based on Two-Layer Fuzzy SVR-TS Model, IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2019, In Press.
        [30]        G. Acampora, A. Vitiello, Interoperable neuro-fuzzy services for emotion-aware ambient intelligence, Neurocomputing, 122 (2013) 3-12.
        [31]        M. Imani, G. A. Montazer, GLCM features and fuzzy nearest neighbor classifier for emotion recognition from face, 2017 7th International Conference on Computer and Knowledge Engineering (ICCKE), Mashhad (2017) 8-13.
        [32]        M. S. Bartlett, G. Littlewort, I. Fasel, and J. R. Movellan, Real time face detection and facial expression recognition: Development and applications to human computer interaction, Computer Vision and Pattern Recognition Workshop (CVPRW ’03), 5 (2003) 53.
        [33]        E. Sariyanidi, H. Gunes, A. Cavallaro, Learning Bases of Activity for Facial Expression Recognition, IEEE Transactions on Image Processing, 26 (4) (2017) 1965-1978.
        [34]        M. Imani, H. Ghassemian, GLCM, Gabor, and Morphology Profiles Fusion for Hyperspectral Image Classification, IEEE proceedings of the 24th Iranian Conference on Electrical Engineering (ICEE 2016), Shiraz, Iran (2016) 460-465.
        [35]        S. Gupta, R. K. Singh, Mathematical morphology based face segmentation and facial feature extraction for facial expression recognition, 2015 International Conference on Futuristic Trends on Computational Analysis and Knowledge Management (ABLAZE), Noida (2015) 691-695.
        [36]        M. Imani, H. Ghassemian, Edge Patch Image-Based Morphological Profiles for Classification of Multispectral and Hyperspectral Data, IET Image Processing, 11 (3) (2017) 164-172.
        [37]        M. Imani, G. A. Montazer, Face Recognition Using Morphological Profile and Feature Space Discriminant Analysis, IEEE proceedings of the 25th Iranian Conference on Electrical Engineering (ICEE 2017), Tehran, Iran (2017) 1729-1734.
        [38]        R. A. Marklein, M. W. Klinker, K. A. Drake, H. G. Polikowsky, E. C. Lessey-Morillon, S. R. Bauer, Morphological profiling using machine learning reveals emergent subpopulations of interferon-γ–stimulated mesenchymal stromal cells that predict immunosuppression, Cytotherapy, 21 (1) (2019) 17-31.
        [39]        P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, Computer Vision and Pattern Recognition, (2001) 511-518.
        [40]        M. J. Lyons, S. Akemastu, M. Kamachi, J. Gyoba, Coding Facial Expressions with Gabor Wavelets, 3rd IEEE International Conference on Automatic Face and Gesture Recognition (1998) 200-205.
        [41]         F. Mirzapour, H. Ghassemian, Using GLCM and Gabor filters for classification of PAN images, 21st Iranian Conference on Electrical Engineering (ICEE 2013), Mashhad, Iran (2013).
 
AUT Journal of Modeling and Simulation
Volume 51, Issue 1
June 2019
Pages 45-56

Files

Share

How to cite

Statistics