Deep Learning-Based Image Captioning for Visual Impairment Using a VGG16 and LSTM Approach

Muhammad Talha Jahangir ()
Additional contact information
Muhammad Talha Jahangir: Department of Computer Science, MNS-University of Engineering and Technology, Multan, Pakistan

International Journal of Innovations in Science & Technology, 2024, vol. 6, issue 4, 1808-1825

Abstract: Visually impaired people face the challenge of gatheringinformation abouttheir surroundings. They are unable to make sense of visually presented information such as capturing images, reading sign boards, movingaround especially when they are alone,or recognizingobjects. This work proposes a novel approach for creating image captioning using two models, one is Convolutional Neural Networks Architectures (VGG16 and ResNet50), and the second is Long Short-Term Memory (LSTM). Using data augmentation and transfer learning on a custom dataset for this work, the system generated accurate image captionsthatincludes a text-to-speech tool that will offer to read back responses to those who are blind or have low vision. The model showed excellent results in training with an accuracy of 90.16 %, and a validation loss of 17.66 %. In caption generation, it obtained the BLEU value ranging from 0.7788 to 0.1 indicating varied caption quality. In general, the average of the Accuracy and Loss results confirms the effectiveness of combining CNNs and LSTMs for improving image descriptions. Thissystem generates such described robust environment for a visually impaired person that it can give the person more freedom to move around and interact with the environment.

Keywords: Image Captioning; Visually Impaired; Convolutional Neural Networks (CNN); Long Short-Term Memory (LSTM); Text-to-Speech; Bilingual Evaluation Understudy (BLEU) Score. (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journal.50sea.com/index.php/IJIST/article/view/1081/1625 (application/pdf)
https://journal.50sea.com/index.php/IJIST/article/view/1081 (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:abq:ijist1:v:6:y:2024:i:4:p:1808-1825

Access Statistics for this article

International Journal of Innovations in Science & Technology is currently edited by Prof. Dr. Syed Amer Mahmood

More articles in International Journal of Innovations in Science & Technology from 50sea
Bibliographic data for series maintained by Iqra Nazeer ().