An open-source MP + CNN + BiLSTM model-based hybrid model for recognizing sign language on smartphones

Hayder M. A. Ghanimi (), Sudhakar Sengan (), Vijaya Bhaskar Sadu (), Parvinder Kaur (), Manju Kaushik (), Roobaea Alroobaea (), Abdullah M. Baqasah (), Majed Alsafyani () and Pankaj Dadheech ()
Additional contact information
Hayder M. A. Ghanimi: University of Warith Al-Anbiyaa
Sudhakar Sengan: PSN College of Engineering and Technology
Vijaya Bhaskar Sadu: Jawaharlal Nehru Technological University
Parvinder Kaur: Chandigarh College of Engineering and Technology, Degree Wing
Manju Kaushik: Amity University Rajasthan
Roobaea Alroobaea: Taif University
Abdullah M. Baqasah: Taif University
Majed Alsafyani: Taif University
Pankaj Dadheech: Swami Keshvanand Institute of Technology, Management and Gramothan (SKIT)

International Journal of System Assurance Engineering and Management, 2024, vol. 15, issue 8, No 17, 3794-3806

Abstract: Abstract The communication barriers experienced by deaf and hard-of-hearing individuals often lead to social isolation and limited access to essential services, underlining a critical need for effective and accessible solutions. Recognizing the unique challenges this community faces—such as the scarcity of sign language interpreters, particularly in remote areas, and the lack of real-time translation tools. This paper proposes the development of a smartphone-runnable sign language recognition model to address the communication problems faced by deaf and hard-of-hearing persons. This proposed model combines Mediapipe hand tracking with particle filtering (PF) to accurately detect and track hand movements, and a convolutional neural network (CNN) and bidirectional long short-term memory based gesture recognition model to model the temporal dynamics of Sign Language gestures. These models use a small number of layers and filters, depthwise separable convolutions, and dropout layers to minimize the computational costs and prevent overfitting, making them suitable for smartphone implementation. This article discusses the existing challenges handled by the deaf and hard-of-hearing community and explains how the proposed model could help overcome these challenges. A MediaPipe + PF model performs feature extraction from the image and data preprocessing. During training, with fewer activation functions and parameters, this proposed model performed better to other CNN with RNN variant models (CNN + LSTM, CNN + GRU) used in the experiments of convergence speed and learning efficiency.

Keywords: CNN; Sign language recognition; Gesture recognition model; BiLSTM; hand gesture recognition; Mobile app (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
http://link.springer.com/10.1007/s13198-024-02376-x Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:ijsaem:v:15:y:2024:i:8:d:10.1007_s13198-024-02376-x

Ordering information: This journal article can be ordered from
http://www.springer.com/engineering/journal/13198

DOI: 10.1007/s13198-024-02376-x

Access Statistics for this article

International Journal of System Assurance Engineering and Management is currently edited by P.K. Kapur, A.K. Verma and U. Kumar

More articles in International Journal of System Assurance Engineering and Management from Springer, The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().