Generative pre-trained transformer based on image content and user personality for caption creation in social media

Ikhsan Ariansyah () and Shintami Chusnul Hidayati ()

Edelweiss Applied Science and Technology, 2024, vol. 8, issue 6, 7366-7385

Abstract: Social media is a platform for sharing information and interactions between users, often involving images with captions that reflect the user's personality. Each user creates distinct captions based on personal traits, making a personalized caption generator beneficial. Currently, existing social media caption generators have limitations, such as requiring payment for full features, lack of support for Bahasa (Indonesian Language), dependency on user input to generate captions, and suboptimal object detection accuracy. To address these issues, a new method is proposed for generating social media captions based on image content and user personality to simplify the caption creation process. This caption generator will be optimized in Bahasa. The content of the image will be explored through image objects and scenery. Image objects are identified using a Graph Convolutional Network (GCN) for personality classification. At the same time, a Convolutional Neural Network (CNN) approach will be employed to detect objects within images, and VGG16 will be used to detect scenery. Then, these three models are combined with a GPT to generate new captions. The model will be trained on public datasets, and subjective evaluation will be used for testing. The outcome of this research is expected to produce relevant captions based on the user's personality, making the captioning process more efficient and relevant to the personality.

Keywords: Caption generation; Generative pre-trained transformer; Object identification; Personality identification; Scene recognition; Social media. (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://learning-gate.com/index.php/2576-8484/article/view/3601/1346 (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:ajp:edwast:v:8:y:2024:i:6:p:7366-7385:id:3601

Access Statistics for this article

More articles in Edelweiss Applied Science and Technology from Learning Gate
Bibliographic data for series maintained by Melissa Fernandes ().