Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs

Ahmed Mateen Buttar, Usama Ahmad, Abdu H. Gumaei (), Adel Assiri, Muhammad Azeem Akbar () and Bader Fahad Alkhamees
Additional contact information
Ahmed Mateen Buttar: Department of Computer Science, University of Agriculture Faisalabad, Faisalabad 38000, Pakistan
Usama Ahmad: Department of Computer Science, University of Agriculture Faisalabad, Faisalabad 38000, Pakistan
Abdu H. Gumaei: Department of Computer Science, College of Computer Engineering and Sciences, Prince Sattam bin Abdulaziz University, Al-Kharj 11942, Saudi Arabia
Adel Assiri: Management Information Systems Department, College of Business, King Khalid University, Abha 61421, Saudi Arabia
Muhammad Azeem Akbar: Software Engineering Department, LUT University, 15210 Lahti, Finland
Bader Fahad Alkhamees: Department of Information Systems, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia

Mathematics, 2023, vol. 11, issue 17, 1-20

Abstract: A speech impairment limits a person’s capacity for oral and auditory communication. A great improvement in communication between the deaf and the general public would be represented by a real-time sign language detector. This work proposes a deep learning-based algorithm that can identify words from a person’s gestures and detect them. There have been many studies on this topic, but the development of static and dynamic sign language recognition models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers’ speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. For the accurate and effective recognition of signs, this study uses two different deep learning-based approaches. We create a real-time American Sign Language detector using the skeleton model, which reliably categorizes continuous signs in sign language in most cases using a deep learning approach. In the second deep learning approach, we create a sign language detector for static signs using YOLOv6. This application is very helpful for sign language users and learners to practice sign language in real time. After training both algorithms separately for static and continuous signs, we create a single algorithm using a hybrid approach. The proposed model, consisting of LSTM with MediaPipe holistic landmarks, achieves around 92% accuracy for different continuous signs, and the YOLOv6 model achieves 96% accuracy over different static signs. Throughout this study, we determine which approach is best for sequential movement detection and for the classification of different signs according to sign language and shows remarkable accuracy in real time.

Keywords: You Only Look Once (YOLO); Long Short-Term Memory (LSTM); deep learning; confusion matrix; convolutional neural network (CNN); MediaPipe holistic (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2023
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (1)

Downloads: (external link)
https://www.mdpi.com/2227-7390/11/17/3729/pdf (application/pdf)
https://www.mdpi.com/2227-7390/11/17/3729/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:11:y:2023:i:17:p:3729-:d:1228999

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().