Recognizing Images and Extracting Useful Inferences by Asking Large Language Models Simple Questions

Nektarios Ioannis Kontolaimakis and Nicholas Panagiotis Sgouros
Additional contact information
Nektarios Ioannis Kontolaimakis: Laboratory Center of Neapolis Lasithiou, Greece
Nicholas Panagiotis Sgouros: University of West Attica, Greece

European Journal of Engineering and Technology Research, 2024, 69-79

Abstract: Laboratory exercises are an essential component of engineering education while the increasing trend towards distance learning presents unique challenges in replicating hands-on experiences. A number of AI-driven solutions have been proposed to facilitate remote laboratory exercises, however the emergence of Multimodal Large Language Models offers novel possibilities for visual recognition in remote settings. Vision AI, a subfield of artificial intelligence, enhances LLM capabilities by allowing them to process visual data through tasks like image recognition and segmentation, making it particularly relevant for use in educational applications. This work evaluates the integration of Vision AI into LLMs like OpenAI's GPT-4 and Anthropic's Claude 3.5 Sonnet, examining their ability to recognize and understand images from laboratory devices such as displays, gauges, and control panels. Our study focuses on the estimation of the performance of GPT-4 and Claude 3.5 Sonnet in laboratory-related image recognition tasks, with results indicating similar high text recognition accuracy (92% for GPT-4 and 91% for Claude 3.5). Despite these successes, challenges persist in spatial awareness and object identification, which are critical for accurate interpretation of complex lab environments. These findings highlight the potential of Vision AI to support remote laboratory exercises, improve accessibility for students in geographically distributed settings, or students with disabilities, and enhance interactive learning tools in STEM education. Future work will focus on refining these capabilities through custom LLM development, advanced prompt engineering, and multimodal approaches, aiming to create more versatile and effective educational technologies for remote and hybrid learning environments.

Keywords: AI; Image Recognition; Large Language Models; Vision AI (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://eu-opensci.org/index.php/ejeng/article/view/63233 Abstract page (text/html)
https://eu-opensci.org/index.php/ejeng/article/download/63233/13247 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:epw:ejeng0:y:2024:id:63233

DOI: 10.24018/ejeng.2024.1.CIE.3233

Access Statistics for this article

More articles in European Journal of Engineering and Technology Research from European Open Science
Bibliographic data for series maintained by Support ().