A Full-Stack Multimodal Assistive Communication System

Shivangi Jindal and Kajal Kori

International Journal of Scientific Research in Science and Technology, 2026, vol. 13, issue 2, 871-876

Abstract: MANUSCRIPT is a full-stack web-based assistive communication platform designed to bridge the communication gap between sign-language / visually impaired users and the hearing/ seeing population. The system enables real-time conversion of hand gestures and spoken language into readable text using computer vision and speech recognition technologies. It integrates MediaPipe Hands for gesture detection and the Web Speech API for speech-to-text conversion, supported by a Flask backend for secure data storage, authentication, and scalability. The primary problem addressed is the lack of accessible, scalable, and affordable communication tools for individuals with hearing, visual or speech impairments. Existing solutions are either limited to single modalities or rely heavily on server-based systems, leading to latency and privacy concerns. The objective of this project is to develop a hybrid system that combines real-time client-side processing with backend support to enhance usability, scalability, and data persistence. The system achieves an average response time of 0.17s for gesture recognition and 0.8s for speech-to-text conversion, enabling near real-time communication. The expected outcome is a robust, user-friendly platform capable of facilitating seamless communication using gestures and speech, with potential applications in education, healthcare, and everyday interactions.

Keywords: Assistive Communication Systems; Gesture Recognition using Computer Vision; Speech-to-Text Conversion (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrst.com/home/article/view/IJSRST2613366 Abstract page (text/html)
https://ijsrst.com/home/article/download/IJSRST2613366/IJSRST2613366 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:etm:ijsrst:v13:y2026:i2:id:1528

DOI: 10.32628/IJSRST2613366

Access Statistics for this article

More articles in International Journal of Scientific Research in Science and Technology from Technoscience Academy
Bibliographic data for series maintained by Pankaj Sharma ().