Non-Manual Gesture Recognition using Transfer Learning Approach

Sameena Javaid ()
Additional contact information
Sameena Javaid: Department of Computer Sciences, School of Engineering and Applied Sciences, Bahria University Karachi Campus, Karachi, Pakistan

International Journal of Innovations in Science & Technology, 2023, vol. 5, issue 4, 576-591

Abstract: Individuals with limited hearing and speech rely on sign language as a fundamental nonverbal mode of communication. It communicates using hand signs, yet the complexity of this mode of expression extends beyond hand movements. Body language and facial expressions are also important in delivering the entire information. While manual (hand movements) and non-manual (facial expressions and body movements) gestures in sign language are important for communication, this field of research has not been substantially investigated, owing to a lack of comprehensive datasets. The current study presents a novel dataset that includes both manual and non-manual gestures in the context of Pakistan Sign Language (PSL). This newly produced dataset consists of. MP4 format films containing seven unique motions involving emotive facial expressions and accompanying hand signs. The dataset was recorded by 100 people. Aside from sign language identification, the dataset opens up possibilities for other applications such as facial expressions, facial feature detection, gender and age classification. In this current study,we evaluated our newly developed dataset for facial expression assessment (non-manual gestures) by YOLO-Face detection methodology successfully extracts faces as Regions of Interest (RoI), with an astounding 90.89% accuracy and an average loss of 0.34. Furthermore, we have used Transfer Learning (TL) using VGG16 architecture to classify seven basic facial expressions and succeeded with 100% accuracy. Insummary, our study produced two different datasets, one with manual and non-manual sign language gestures, the second with Asian faces to find seven basic facial expressions. With both the dataset,our validation techniques found promising results.

Keywords: Face Expressions; Non-Manual Gestures; Pakistan Sign Language; Region of Interest; YOLO-Face; Transfer Learning; VGG16 (search for similar items in EconPapers)
Date: 2023
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journal.50sea.com/index.php/IJIST/article/view/577/1110 (application/pdf)
https://journal.50sea.com/index.php/IJIST/article/view/577 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:abq:ijist1:v:5:y:2023:i:4:p:576-591

Access Statistics for this article

International Journal of Innovations in Science & Technology is currently edited by Prof. Dr. Syed Amer Mahmood

More articles in International Journal of Innovations in Science & Technology from 50sea
Bibliographic data for series maintained by Iqra Nazeer ().