UPM Institutional Repository

Leveraging transfer learning for spatio-temporal human activity recognition from video sequences


Citation

Muneer Butt, Umair and Aman Ullah, Hadiqa and Letchmunan, Sukumar and Tariq, Iqra and Hafinaz Hassan, Fadratul and Wei Koh, Tieng (2023) Leveraging transfer learning for spatio-temporal human activity recognition from video sequences. Computers, Materials and Continua, 74 (3). pp. 5017-5033. ISSN 1546-2218; eISSN: 1546-2226

Abstract

Human Activity Recognition (HAR) is an active research area due to its applications in pervasive computing, human-computer interaction, artificial intelligence, health care, and social sciences.Moreover, dynamic environments and anthropometric differences between individuals make it harder to recognize actions. This study focused on human activity in video sequences acquired with an RGB camera because of its vast range of real-world applications. It uses two-stream ConvNet to extract spatial and temporal information and proposes a fine-tuned deep neural network. Moreover, the transfer learning paradigm is adopted to extract varied and fixed frames while reusing object identification information. Six state-of-the-art pre-trained models are exploited to find the best model for spatial feature extraction. For temporal sequence, this study uses dense optical flow following the two-stream ConvNet and Bidirectional Long Short TermMemory (BiLSTM) to capture longtermdependencies. Two state-of-the-art datasets, UCF101 andHMDB51, are used for evaluation purposes. In addition, seven state-of-the-art optimizers are used to fine-tune the proposed network parameters. Furthermore, this study utilizes an ensemble mechanism to aggregate spatial-temporal features using a four-stream Convolutional Neural Network (CNN), where two streams use RGB data. In contrast, the other uses optical flow images. Finally, the proposed ensemble approach using max hard voting outperforms state-ofthe- art methods with 96.30 and 90.07 accuracies on the UCF101 and HMDB51 datasets.


Download File

[img] Text
TSP_CMC_35512.pdf - Published Version

Download (919kB)
Official URL or Download Paper: https://www.techscience.com/cmc/v74n3/50975

Additional Metadata

Item Type: Article
Divisions: Faculty of Computer Science and Information Technology
DOI Number: https://doi.org/10.32604/cmc.2023.035512
Publisher: Tech Science Press
Keywords: Human activity recognition; Deep learning; Transfer learning; Neural network; Ensemble learning; Spatio-temporal; Sustainable cities and communities
Depositing User: Ms. Nur Aina Ahmad Mustafa
Date Deposited: 17 Dec 2024 04:00
Last Modified: 17 Dec 2024 04:00
Altmetrics: http://www.altmetric.com/details.php?domain=psasir.upm.edu.my&doi=10.32604/cmc.2023.035512
URI: http://psasir.upm.edu.my/id/eprint/109555
Statistic Details: View Download Statistic

Actions (login required)

View Item View Item