Remote elderly healthcare: a robust deep learning approach for wearable sensors-based complex activities recognition

Document Type : Research Article


1 Department of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran

2 Department of Computer Engineering, Faculty of Engineering, Alzahra University, Tehran, Iran

3 Department of Electrical Engineering, Amirkabir University of Technology, Tehran, Iran


Human activity recognition system (HARS) is critical for monitoring individuals' health and movements, especially the elderly and the disabled, in different environments. Wearable sensors are valuable tools for information acquisition due to the mobility of people. Researchers have proposed different methods for sensor-based HAR, which face challenges. This paper uses deep learning (DL), which combines the deep convolution neural network (CNN) and long short-term memory (DeepConvLSTM) to extract and select features to recognize high-performance activity. Then, the Softmax-Support Vector Machine (SofSVM) performs the classification and recognition operations. This paper uses a comprehensive nested pipe and filter (NPF) architecture. Filters are usually independent, but some filters can be bidirectional to improve the performance of complex activity recognition. There is two-way communication between convolutional layers and long short-term memory (LSTM). The Opportunity dataset is public and includes complex activities. The results with this dataset display that the proposed work improves the weighting F-measure of HAR. This paper has also shown other experiments; that involve comparing the number of sensors, max-pooling size, and convolutional filter size. According to this dataset, the proposed method weighting F-measure is 0.929. The proposed approach is indeed effective in activity recognition, and the NPF architecture covers all the components of the activity recognition process.


Main Subjects

  1. 1. Keyvanpour, MR., Khanbani, N., Aliniya, Z. (2021). Detection of individual activities in video sequences based on fast interference discovery and semi-supervised method. Multimedia Tools and Applications, 80, 13879-13910.‏

    1. Kharati, E., Khalily-Dermany, M., & Kermajani, H. (2019). Increasing the Value of Collected Data and Reducing Energy Consumption using Network Coding and Mobile Sinks in Wireless Sensor Networks. AUT Journal of Modeling and Simulation51(1), 3-14.‏
    2. Ezatzadeh, S., Keyvanpour, M. R., & Shojaedini, S. V. (2021). A human fall detection framework based on multi-camera fusion. Journal of Experimental & Theoretical Artificial Intelligence, 1-20.‏
    3. Serpush, F., Menhaj, M. B., Masoumi, B., & Karasfi, B. (2022). Wearable Sensor-Based Human Activity Recognition in the Smart Healthcare System. Computational Intelligence and Neuroscience2022.‏
    4. Hassan, M. M., Huda, S., Uddin, M. Z., Almogren, A., & Alrubaian, M. (2018). Human activity recognition from body sensor data using deep learning. Journal of medical systems42(6), 1-8.‏
    5. Xu, S., Zhang, L., Huang, W., Wu, H., & Song, A. (2022). Deformable Convolutional Networks for Multimodal Human Activity Recognition Using Wearable Sensors. IEEE Transactions on Instrumentation and Measurement71, 1-14.‏
    6. Qi, W., Su, H., Yang, C., Ferrigno, G., De Momi, E., & Aliverti, A. (2019). A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone. Sensors19(17), 3731.‏
    7. Kuncan, F., Kaya, Y., Tekin, R., & Kuncan, M. (2022). A new approach for physical human activity recognition based on co-occurrence matrices. The Journal of Supercomputing78(1), 1048-1070.‏
    8. Ahad, M. A. R., Antar, A. D., & Ahmed, M. (2021). Deep learning for sensor-based activity recognition: recent trends. IoT Sensor-Based Activity Recognition, 149-173.‏
    9. Razzaq, M. A., Cleland, I., Nugent, C., & Lee, S. (2020). SemImput: Bridging Semantic Imputation with Deep Learning for Complex Human Activity Recognition. Sensors20(10), 2771.‏
    10. Suto, J., Oniga, S., Lung, C., & Orha, I. (2020). Comparison of offline and real-time human activity recognition results using machine learning techniques. Neural computing and applications32(20), 15673-15686.‏
    11. Miranda, L., Viterbo, J., & Bernardini, F. (2022). A survey on the use of machine learning methods in context-aware middlewares for human activity recognition. Artificial Intelligence Review55(4), 3369-3400.‏
    12. Sun, J., Fu, Y., Li, S., He, J., Xu, C., & Tan, L. (2018). Sequential human activity recognition based on deep convolutional network and extreme learning machine using wearable sensors. Journal of Sensors2018.‏
    13. de la Concepción, M. Á. Á., Morillo, L. M. S., García, J. A. Á., & González-Abril, L. (2017). Mobile activity recognition and fall detection system for elderly people using Ameva algorithm. Pervasive and Mobile Computing34, 3-13.‏
    14. Alemdar, H., Van Kasteren, T. L. M., & Ersoy, C. (2017). Active learning with uncertainty sampling for large scale activity recognition in smart homes. Journal of Ambient Intelligence and Smart Environments9(2), 209-223.‏
    15. Kerdjidj, O., Ramzan, N., Ghanem, K., Amira, A., & Chouireb, F. (2020). Fall detection and human activity classification using wearable sensors and compressed sensing. Journal of Ambient Intelligence and Humanized Computing11(1), 349-361.‏
    16. Inoue, M., Inoue, S., & Nishida, T. (2018). Deep recurrent neural network for mobile human activity recognition with high throughput. Artificial Life and Robotics23(2), 173-185.‏
    17. Serpush, F., & Rezaei, M. (2021). Complex human action recognition using a hierarchical feature reduction and deep learning-based method. SN Computer Science2(2), 1-15.‏
    18. Rajabi, M., & Khaloozadeh, H. (2020). Long-term prediction in Tehran stock market using a new architecture of Deep neural networks. AUT Journal of Modeling and Simulation52(2), 4-4.‏
    19. Mukherjee, D., Mondal, R., Singh, P. K., Sarkar, R., & Bhattacharjee, D. (2020). EnsemConvNet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications. Multimedia Tools and Applications79(41), 31663-31690.‏
    20. Hassan, M. M., Ullah, S., Hossain, M. S., & Alelaiwi, A. (2021). An end-to-end deep learning model for human activity recognition from highly sparse body sensor data in internet of medical things environment. The Journal of Supercomputing77(3), 2237-2250.‏
    21. Thakur, D., & Biswas, S. (2021). Feature fusion using deep learning for smartphone based human activity recognition. International Journal of Information Technology13(4), 1615-1624.‏
    22. Hammerla, N. Y., Halloran, S., & Plötz, T. (2016, July). Deep, convolutional, and recurrent models for human activity recognition using wearables. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence(pp. 1533-1540).‏
    23. Ronao, C. A., & Cho, S. B. (2016). Human activity recognition with smartphone sensors using deep learning neural networks. Expert systems with applications59, 235-244.‏
    24. De Vita, A., Russo, A., Pau, D., Di Benedetto, L., Rubino, A., & Licciardo, G. D. (2020). A partially binarized hybrid neural network system for low-power and resource constrained human activity recognition. IEEE Transactions on Circuits and Systems I: Regular Papers67(11), 3893-3904.‏
    25. Guan, Y., & Plötz, T. (2017). Ensembles of deep lstm learners for activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies1(2), 1-28.‏
    26. Wang, L. (2016). Recognition of human activities using continuous autoencoders with wearable sensors. Sensors16(2), 189.‏
    27. Munoz-Organero, M., & Ruiz-Blazquez, R. (2017). Time-elastic generative model for acceleration time series in human activity recognition. Sensors17(2), 319.‏
    28. Semwal, V. B., Gupta, A., & Lalwani, P. (2021). An optimized hybrid deep learning model using ensemble learning approach for human walking activities recognition. The Journal of Supercomputing77(11), 12256-12279.‏
    29. Paraschiakos, S., Cachucho, R., Moed, M., van Heemst, D., Mooijaart, S., Slagboom, E. P., ... & Beekman, M. (2020). Activity recognition using wearable sensors for tracking the elderly. User Modeling and User-Adapted Interaction30(3), 567-605.‏
    30. Hassan, M. M., Uddin, M. Z., Mohamed, A., & Almogren, A. (2018). A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems81, 307-313.‏
    31. Zhao, Y., Yang, R., Chevalier, G., Xu, X., & Zhang, Z. (2018). Deep residual bidir-LSTM for human activity recognition using wearable sensors. Mathematical Problems in Engineering2018.‏
    32. Kim, E. (2020). Interpretable and accurate convolutional neural networks for human activity recognition. IEEE Transactions on Industrial Informatics16(11), 7190-7198.‏
    33. Ordóñez, F. J., & Roggen, D. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors16(1), 115.‏
    34. Jiang, Y., Hernandez, V., Venture, G., Kulić, D., & K. Chen, B. (2021). A data-driven approach to predict fatigue in exercise based on motion data from wearable sensors or force plate. Sensors21(4), 1499.‏
    35. Lee, S. M., Yoon, S. M., & Cho, H. (2017, February). Human activity recognition from accelerometer data using Convolutional Neural Network. In 2017 ieee international conference on big data and smart computing (bigcomp)(pp. 131-134). IEEE.‏
    36. Ronao, C. A., & Cho, S. B. (2015, November). Deep convolutional neural networks for human activity recognition with smartphone sensors. In International Conference on Neural Information Processing(pp. 46-53). Springer, Cham.‏
    37. Wu, D., Wang, Z., Chen, Y., & Zhao, H. (2016). Mixed-kernel based weighted extreme learning machine for inertial sensor based human activity recognition with imbalanced dataset. Neurocomputing190, 35-49.‏
    38. Hassani, S. A., Mobaraki, H., Bayat, M., & Mafimoradi, S. (2013). Right place of human resource management in the reform of health sector. Iranian journal of public health42(1), 56.‏
    39. Davila, J. C., Cretu, A. M., & Zaremba, M. (2017). Wearable sensor data classification for human activity recognition based on an iterative learning framework. Sensors17(6), 1287.‏
    40. Bazgir, O., Habibi, S. A. H., Palma, L., Pierleoni, P., & Nafees, S. (2018). A classification system for assessment and home monitoring of tremor in patients with Parkinson's disease. Journal of medical signals and sensors8(2), 65.‏
    41. Nazari, J., Fathi, P. S., Sharahi, N., Taheri, M., Amini, P., & Almasi-Hashiani, A. (2022). Evaluating Measles Incidence Rates Using Machine Learning and Time Series Methods in the Center of Iran, 1997–2020. Iranian Journal of Public Health51(4), 904.‏
    42. Muzammal, M., Talat, R., Sodhro, A. H., & Pirbhulal, S. (2020). A multi-sensor data fusion enabled ensemble approach for medical data from body sensor networks. Information Fusion53, 155-164.‏
    43. Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2021). Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. ACM Computing Surveys (CSUR)54(4), 1-40.‏
    44. Peng, L., Chen, L., Ye, Z., & Zhang, Y. (2018). Aroma: A deep multi-task learning based simple and complex human activity recognition method using wearable sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies2(2), 1-16.‏
    45. Davila, J. C., Cretu, A. M., & Zaremba, M. (2017). Wearable sensor data classification for human activity recognition based on an iterative learning framework. Sensors, 17(6), 1287.‏
    46. Yao, R., Lin, G., Shi, Q., & Ranasinghe, D. C. (2018). Efficient dense labelling of human activity sequences from wearables using fully convolutional networks. Pattern Recognition78, 252-266.‏
    47. Kautz, T., Groh, B. H., Hannink, J., Jensen, U., Strubberg, H., & Eskofier, B. M. (2017). Activity recognition in beach volleyball using a deep convolutional neural network. Data Mining and Knowledge Discovery31(6), 1678-1705.‏
    48. Suto, J., & Oniga, S. (2018). Efficiency investigation of artificial neural networks in human activity recognition. Journal of Ambient Intelligence and Humanized Computing9(4), 1049-1060.‏
    49. Liu, H., & Wang, L. (2018). Gesture recognition for human-robot collaboration: A review. International Journal of Industrial Ergonomics68, 355-367.‏
    50. Kang, J., Kim, J., Lee, S., & Sohn, M. (2020). Transition activity recognition using fuzzy logic and overlapped sliding window-based convolutional neural networks. The Journal of Supercomputing76(10), 8003-8020.‏
    51. Abedini, F., Menhaj, M. B., & Keyvanpour, M. R. (2019). An MLP-based representation of neural tensor networks for the RDF data models. Neural Computing and Applications31(2), 1135-1144.‏
    52. Shu, X., Zhang, L., Sun, Y., & Tang, J. (2020). Host–parasite: Graph LSTM-in-LSTM for group activity recognition. IEEE transactions on neural networks and learning systems32(2), 663-674.‏
    53. Reining, C., Niemann, F., Moya Rueda, F., Fink, G. A., & ten Hompel, M. (2019). Human activity recognition for production and logistics—a systematic literature review. Information10(8), 245.‏
    54. Taha, Z., Musa, R. M., Majeed, A. P. A., Alim, M. M., & Abdullah, M. R. (2018). The identification of high potential archers based on fitness and motor ability variables: A Support Vector Machine approach. Human movement science57, 184-193.‏
    55. Serpush, F., & Keyvanpour, M. (2014). QEA: a new systematic and comprehensive classification of query expansion approaches. Journal of Computer & Robotics7(1), 1-17.‏
    56. Chen, Z., Zhu, Q., Soh, Y. C., & Zhang, L. (2017). Robust human activity recognition using smartphone sensors via CT-PCA and online SVM. IEEE transactions on industrial informatics13(6), 3070-3080.‏
    57. Erdaş, Ç. B., & Güney, S. (2021). Human activity recognition by using different deep learning approaches for wearable sensors. Neural Processing Letters53(3), 1795-1809.‏
    58. Tran, D. N., & Phan, D. D. (2016, January). Human activities recognition in android smartphone using support vector machine. In 2016 7th international conference on intelligent systems, modelling and simulation (isms)(pp. 64-68). IEEE.‏
    59. Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013). The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters34(15), 2033-2042.‏
    60. Li, F., Shirahama, K., Nisar, M. A., Köping, L., & Grzegorzek, M. (2018). Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors18(2), 679.‏
    61. Tuncer, T., & Ertam, F. (2021). Novel tent pooling based human activity recognition approach. Multimedia Tools and Applications80(3), 4639-4653.‏
    62. Zebin, T., Scully, P. J., Peek, N., Casson, A. J., & Ozanyan, K. B. (2019). Design and implementation of a convolutional neural network on an edge computing smartphone for human activity recognition. IEEE Access7, 133509-133520.‏
    63. Yao, R., Lin, G., Shi, Q., & Ranasinghe, D. C. (2018). Efficient dense labelling of human activity sequences from wearables using fully convolutional networks. Pattern Recognition78, 252-266.‏
    64. Esmaeili, V., Mohassel Feghhi, M., & Shahdi, S. O. (2022). Automatic Micro-Expression Recognition using LBP-SIPl and FR-CNN. AUT Journal of Modeling and Simulation54(1), 5-5.‏
    65. Shojaedini, S. V., Morabbi, S., & Keyvanpour, M. (2018). A new method for detecting p300 signals by using deep learning: hyperparameter tuning in high-dimensional space by minimizing nonconvex error function. Journal of medical signals and sensors8(4), 205.‏
    66. Wang, L., & Liu, R. (2020). Human activity recognition based on wearable sensor using hierarchical deep LSTM networks. Circuits, Systems, and Signal Processing39(2), 837-856.‏
    67. Keyvanpour, M., & Serpush, F. (2019). ESLMT: a new clustering method for biomedical document retrieval. Biomedical Engineering/Biomedizinische Technik64(6), 729-741.‏
    68. Zeng, M., Nguyen, L. T., Yu, B., Mengshoel, O. J., Zhu, J., Wu, P., & Zhang, J. (2014, November). Convolutional neural networks for human activity recognition using mobile sensors. In 6th international conference on mobile computing, applications and services(pp. 197-205). IEEE.‏
    69. Guo, Y., Wu, Z., & Ji, Y. (2017, August). A hybrid deep representation learning model for time series classification and prediction. In 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM)(pp. 226-231). IEEE.‏