AUT Journal of Modeling and Simulation

AUT Journal of Modeling and Simulation

Remote elderly healthcare: a robust deep learning approach for wearable sensors-based complex activities recognition

Document Type : Research Article

Authors
Fatemeh Serpush 1
Mohammadreza Keyvanpour 2
Mohammad bagher Menhaj 3
1 Department of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran
2 Department of Computer Engineering, Faculty of Engineering, Alzahra University, Tehran, Iran
3 Department of Electrical Engineering, Amirkabir University of Technology, Tehran, Iran
10.22060/miscj.2023.21984.5308
Abstract
Human activity recognition system (HARS) is critical for monitoring individuals' health and movements, especially the elderly and the disabled, in different environments. Wearable sensors are valuable tools for information acquisition due to the mobility of people. Researchers have proposed different methods for sensor-based HAR, which face challenges. This paper uses deep learning (DL), which combines the deep convolution neural network (CNN) and long short-term memory (DeepConvLSTM) to extract and select features to recognize high-performance activity. Then, the Softmax-Support Vector Machine (SofSVM) performs the classification and recognition operations. This paper uses a comprehensive nested pipe and filter (NPF) architecture. Filters are usually independent, but some filters can be bidirectional to improve the performance of complex activity recognition. There is two-way communication between convolutional layers and long short-term memory (LSTM). The Opportunity dataset is public and includes complex activities. The results with this dataset display that the proposed work improves the weighting F-measure of HAR. This paper has also shown other experiments; that involve comparing the number of sensors, max-pooling size, and convolutional filter size. According to this dataset, the proposed method weighting F-measure is 0.929. The proposed approach is indeed effective in activity recognition, and the NPF architecture covers all the components of the activity recognition process.
Keywords
Complex human activity
convolutional neural networks
Long Short-term Memory
Wearable Sensors
Support Vector Machine
Subjects

  1. 1. Keyvanpour, MR., Khanbani, N., Aliniya, Z. (2021). Detection of individual activities in video sequences based on fast interference discovery and semi-supervised method. Multimedia Tools and Applications, 80, 13879-13910.‏

    1. Kharati, E., Khalily-Dermany, M., & Kermajani, H. (2019). Increasing the Value of Collected Data and Reducing Energy Consumption using Network Coding and Mobile Sinks in Wireless Sensor Networks. AUT Journal of Modeling and Simulation, 51(1), 3-14.‏
    2. Ezatzadeh, S., Keyvanpour, M. R., & Shojaedini, S. V. (2021). A human fall detection framework based on multi-camera fusion. Journal of Experimental & Theoretical Artificial Intelligence, 1-20.‏
    3. Serpush, F., Menhaj, M. B., Masoumi, B., & Karasfi, B. (2022). Wearable Sensor-Based Human Activity Recognition in the Smart Healthcare System. Computational Intelligence and Neuroscience, 2022.‏
    4. Hassan, M. M., Huda, S., Uddin, M. Z., Almogren, A., & Alrubaian, M. (2018). Human activity recognition from body sensor data using deep learning. Journal of medical systems, 42(6), 1-8.‏
    5. Xu, S., Zhang, L., Huang, W., Wu, H., & Song, A. (2022). Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable Sensors. IEEE Transactions on Instrumentation and Measurement, 71, 1-14.‏
    6. Qi, W., Su, H., Yang, C., Ferrigno, G., De Momi, E., & Aliverti, A. (2019). A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone. Sensors, 19(17), 3731.‏
    7. Kuncan, F., Kaya, Y., Tekin, R., & Kuncan, M. (2022). A new approach for physical human activity recognition based on co-occurrence matrices. The Journal of Supercomputing, 78(1), 1048-1070.‏
    8. Ahad, M. A. R., Antar, A. D., & Ahmed, M. (2021). Deep learning for sensor-based activity recognition: recent trends. IoT Sensor-Based Activity Recognition, 149-173.‏
    9. Razzaq, M. A., Cleland, I., Nugent, C., & Lee, S. (2020). SemImput: Bridging Semantic Imputation with Deep Learning for Complex Human Activity Recognition. Sensors, 20(10), 2771.‏
    10. Suto, J., Oniga, S., Lung, C., & Orha, I. (2020). Comparison of offline and real-time human activity recognition results using machine learning techniques. Neural computing and applications, 32(20), 15673-15686.‏
    11. Miranda, L., Viterbo, J., & Bernardini, F. (2022). A survey on the use of machine learning methods in context-aware middlewares for human activity recognition. Artificial Intelligence Review, 55(4), 3369-3400.‏
    12. Sun, J., Fu, Y., Li, S., He, J., Xu, C., & Tan, L. (2018). Sequential human activity recognition based on deep convolutional network and extreme learning machine using wearable sensors. Journal of Sensors, 2018.‏
    13. de la Concepción, M. Á. Á., Morillo, L. M. S., García, J. A. Á., & González-Abril, L. (2017). Mobile activity recognition and fall detection system for elderly people using Ameva algorithm. Pervasive and Mobile Computing, 34, 3-13.‏
    14. Alemdar, H., Van Kasteren, T. L. M., & Ersoy, C. (2017). Active learning with uncertainty sampling for large scale activity recognition in smart homes. Journal of Ambient Intelligence and Smart Environments, 9(2), 209-223.‏
    15. Kerdjidj, O., Ramzan, N., Ghanem, K., Amira, A., & Chouireb, F. (2020). Fall detection and human activity classification using wearable sensors and compressed sensing. Journal of Ambient Intelligence and Humanized Computing, 11(1), 349-361.‏
    16. Inoue, M., Inoue, S., & Nishida, T. (2018). Deep recurrent neural network for mobile human activity recognition with high throughput. Artificial Life and Robotics, 23(2), 173-185.‏
    17. Serpush, F., & Rezaei, M. (2021). Complex human action recognition using a hierarchical feature reduction and deep learning-based method. SN Computer Science, 2(2), 1-15.‏
    18. Rajabi, M., & Khaloozadeh, H. (2020). Long-term prediction in Tehran stock market using a new architecture of Deep neural networks. AUT Journal of Modeling and Simulation, 52(2), 4-4.‏
    19. Mukherjee, D., Mondal, R., Singh, P. K., Sarkar, R., & Bhattacharjee, D. (2020). EnsemConvNet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications. Multimedia Tools and Applications, 79(41), 31663-31690.‏
    20. Hassan, M. M., Ullah, S., Hossain, M. S., & Alelaiwi, A. (2021). An end-to-end deep learning model for human activity recognition from highly sparse body sensor data in internet of medical things environment. The Journal of Supercomputing, 77(3), 2237-2250.‏
    21. Thakur, D., & Biswas, S. (2021). Feature fusion using deep learning for smartphone based human activity recognition. International Journal of Information Technology, 13(4), 1615-1624.‏
    22. Hammerla, N. Y., Halloran, S., & Plötz, T. (2016, July). Deep, convolutional, and recurrent models for human activity recognition using wearables. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence(pp. 1533-1540).‏
    23. Ronao, C. A., & Cho, S. B. (2016). Human activity recognition with smartphone sensors using deep learning neural networks. Expert systems with applications, 59, 235-244.‏
    24. De Vita, A., Russo, A., Pau, D., Di Benedetto, L., Rubino, A., & Licciardo, G. D. (2020). A partially binarized hybrid neural network system for low-power and resource constrained human activity recognition. IEEE Transactions on Circuits and Systems I: Regular Papers, 67(11), 3893-3904.‏
    25. Guan, Y., & Plötz, T. (2017). Ensembles of deep lstm learners for activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(2), 1-28.‏
    26. Wang, L. (2016). Recognition of human activities using continuous autoencoders with wearable sensors. Sensors, 16(2), 189.‏
    27. Munoz-Organero, M., & Ruiz-Blazquez, R. (2017). Time-elastic generative model for acceleration time series in human activity recognition. Sensors, 17(2), 319.‏
    28. Semwal, V. B., Gupta, A., & Lalwani, P. (2021). An optimized hybrid deep learning model using ensemble learning approach for human walking activities recognition. The Journal of Supercomputing, 77(11), 12256-12279.‏
    29. Paraschiakos, S., Cachucho, R., Moed, M., van Heemst, D., Mooijaart, S., Slagboom, E. P., ... & Beekman, M. (2020). Activity recognition using wearable sensors for tracking the elderly. User Modeling and User-Adapted Interaction, 30(3), 567-605.‏
    30. Hassan, M. M., Uddin, M. Z., Mohamed, A., & Almogren, A. (2018). A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems, 81, 307-313.‏
    31. Zhao, Y., Yang, R., Chevalier, G., Xu, X., & Zhang, Z. (2018). Deep residual bidir-LSTM for human activity recognition using wearable sensors. Mathematical Problems in Engineering, 2018.‏
    32. Kim, E. (2020). Interpretable and accurate convolutional neural networks for human activity recognition. IEEE Transactions on Industrial Informatics, 16(11), 7190-7198.‏
    33. Ordóñez, F. J., & Roggen, D. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1), 115.‏
    34. Jiang, Y., Hernandez, V., Venture, G., Kulić, D., & K. Chen, B. (2021). A data-driven approach to predict fatigue in exercise based on motion data from wearable sensors or force plate. Sensors, 21(4), 1499.‏
    35. Lee, S. M., Yoon, S. M., & Cho, H. (2017, February). Human activity recognition from accelerometer data using Convolutional Neural Network. In 2017 ieee international conference on big data and smart computing (bigcomp)(pp. 131-134). IEEE.‏
    36. Ronao, C. A., & Cho, S. B. (2015, November). Deep convolutional neural networks for human activity recognition with smartphone sensors. In International Conference on Neural Information Processing(pp. 46-53). Springer, Cham.‏
    37. Wu, D., Wang, Z., Chen, Y., & Zhao, H. (2016). Mixed-kernel based weighted extreme learning machine for inertial sensor based human activity recognition with imbalanced dataset. Neurocomputing, 190, 35-49.‏
    38. Hassani, S. A., Mobaraki, H., Bayat, M., & Mafimoradi, S. (2013). Right place of human resource management in the reform of health sector. Iranian journal of public health, 42(1), 56.‏
    39. Davila, J. C., Cretu, A. M., & Zaremba, M. (2017). Wearable sensor data classification for human activity recognition based on an iterative learning framework. Sensors, 17(6), 1287.‏
    40. Bazgir, O., Habibi, S. A. H., Palma, L., Pierleoni, P., & Nafees, S. (2018). A classification system for assessment and home monitoring of tremor in patients with Parkinson's disease. Journal of medical signals and sensors, 8(2), 65.‏
    41. Nazari, J., Fathi, P. S., Sharahi, N., Taheri, M., Amini, P., & Almasi-Hashiani, A. (2022). Evaluating Measles Incidence Rates Using Machine Learning and Time Series Methods in the Center of Iran, 1997–2020. Iranian Journal of Public Health, 51(4), 904.‏
    42. Muzammal, M., Talat, R., Sodhro, A. H., & Pirbhulal, S. (2020). A multi-sensor data fusion enabled ensemble approach for medical data from body sensor networks. Information Fusion, 53, 155-164.‏
    43. Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2021). Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. ACM Computing Surveys (CSUR), 54(4), 1-40.‏
    44. Peng, L., Chen, L., Ye, Z., & Zhang, Y. (2018). Aroma: A deep multi-task learning based simple and complex human activity recognition method using wearable sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(2), 1-16.‏
    45. Davila, J. C., Cretu, A. M., & Zaremba, M. (2017). Wearable sensor data classification for human activity recognition based on an iterative learning framework. Sensors, 17(6), 1287.‏
    46. Yao, R., Lin, G., Shi, Q., & Ranasinghe, D. C. (2018). Efficient dense labelling of human activity sequences from wearables using fully convolutional networks. Pattern Recognition, 78, 252-266.‏
    47. Kautz, T., Groh, B. H., Hannink, J., Jensen, U., Strubberg, H., & Eskofier, B. M. (2017). Activity recognition in beach volleyball using a deep convolutional neural network. Data Mining and Knowledge Discovery, 31(6), 1678-1705.‏
    48. Suto, J., & Oniga, S. (2018). Efficiency investigation of artificial neural networks in human activity recognition. Journal of Ambient Intelligence and Humanized Computing, 9(4), 1049-1060.‏
    49. Liu, H., & Wang, L. (2018). Gesture recognition for human-robot collaboration: A review. International Journal of Industrial Ergonomics, 68, 355-367.‏
    50. Kang, J., Kim, J., Lee, S., & Sohn, M. (2020). Transition activity recognition using fuzzy logic and overlapped sliding window-based convolutional neural networks. The Journal of Supercomputing, 76(10), 8003-8020.‏
    51. Abedini, F., Menhaj, M. B., & Keyvanpour, M. R. (2019). An MLP-based representation of neural tensor networks for the RDF data models. Neural Computing and Applications, 31(2), 1135-1144.‏
    52. Shu, X., Zhang, L., Sun, Y., & Tang, J. (2020). Host–parasite: Graph LSTM-in-LSTM for group activity recognition. IEEE transactions on neural networks and learning systems, 32(2), 663-674.‏
    53. Reining, C., Niemann, F., Moya Rueda, F., Fink, G. A., & ten Hompel, M. (2019). Human activity recognition for production and logistics—a systematic literature review. Information, 10(8), 245.‏
    54. Taha, Z., Musa, R. M., Majeed, A. P. A., Alim, M. M., & Abdullah, M. R. (2018). The identification of high potential archers based on fitness and motor ability variables: A Support Vector Machine approach. Human movement science, 57, 184-193.‏
    55. Serpush, F., & Keyvanpour, M. (2014). QEA: a new systematic and comprehensive classification of query expansion approaches. Journal of Computer & Robotics, 7(1), 1-17.‏
    56. Chen, Z., Zhu, Q., Soh, Y. C., & Zhang, L. (2017). Robust human activity recognition using smartphone sensors via CT-PCA and online SVM. IEEE transactions on industrial informatics, 13(6), 3070-3080.‏
    57. Erdaş, Ç. B., & Güney, S. (2021). Human activity recognition by using different deep learning approaches for wearable sensors. Neural Processing Letters, 53(3), 1795-1809.‏
    58. Tran, D. N., & Phan, D. D. (2016, January). Human activities recognition in android smartphone using support vector machine. In 2016 7th international conference on intelligent systems, modelling and simulation (isms)(pp. 64-68). IEEE.‏
    59. Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013). The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters, 34(15), 2033-2042.‏
    60. Li, F., Shirahama, K., Nisar, M. A., Köping, L., & Grzegorzek, M. (2018). Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors, 18(2), 679.‏
    61. Tuncer, T., & Ertam, F. (2021). Novel tent pooling based human activity recognition approach. Multimedia Tools and Applications, 80(3), 4639-4653.‏
    62. Zebin, T., Scully, P. J., Peek, N., Casson, A. J., & Ozanyan, K. B. (2019). Design and implementation of a convolutional neural network on an edge computing smartphone for human activity recognition. IEEE Access, 7, 133509-133520.‏
    63. Yao, R., Lin, G., Shi, Q., & Ranasinghe, D. C. (2018). Efficient dense labelling of human activity sequences from wearables using fully convolutional networks. Pattern Recognition, 78, 252-266.‏
    64. Esmaeili, V., Mohassel Feghhi, M., & Shahdi, S. O. (2022). Automatic Micro-Expression Recognition using LBP-SIPl and FR-CNN. AUT Journal of Modeling and Simulation, 54(1), 5-5.‏
    65. Shojaedini, S. V., Morabbi, S., & Keyvanpour, M. (2018). A new method for detecting p300 signals by using deep learning: hyperparameter tuning in high-dimensional space by minimizing nonconvex error function. Journal of medical signals and sensors, 8(4), 205.‏
    66. Wang, L., & Liu, R. (2020). Human activity recognition based on wearable sensor using hierarchical deep LSTM networks. Circuits, Systems, and Signal Processing, 39(2), 837-856.‏
    67. Keyvanpour, M., & Serpush, F. (2019). ESLMT: a new clustering method for biomedical document retrieval. Biomedical Engineering/Biomedizinische Technik, 64(6), 729-741.‏
    68. Zeng, M., Nguyen, L. T., Yu, B., Mengshoel, O. J., Zhu, J., Wu, P., & Zhang, J. (2014, November). Convolutional neural networks for human activity recognition using mobile sensors. In 6th international conference on mobile computing, applications and services(pp. 197-205). IEEE.‏
    69. Guo, Y., Wu, Z., & Ji, Y. (2017, August). A hybrid deep representation learning model for time series classification and prediction. In 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM)(pp. 226-231). IEEE.‏
AUT Journal of Modeling and Simulation
Volume 55, Issue 1
June 2023
Pages 109-126