Human–AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era

Rui Yu (), Sooyeon Lee, Jingyi Xie, Syed Masum Billah and John M. Carroll ()
Additional contact information
Rui Yu: Department of Computer Science and Engineering, University of Louisville, Louisville, KY 40208, USA
Sooyeon Lee: Department of Informatics, Ying Wu College of Computing, New Jersey Institute of Technology, Newark, NJ 07102, USA
Jingyi Xie: College of Information Sciences and Technology, Pennsylvania State University, University Park, PA 16802, USA
Syed Masum Billah: College of Information Sciences and Technology, Pennsylvania State University, University Park, PA 16802, USA
John M. Carroll: College of Information Sciences and Technology, Pennsylvania State University, University Park, PA 16802, USA

Future Internet, 2024, vol. 16, issue 7, 1-32

Abstract: Remote sighted assistance (RSA) has emerged as a conversational technology aiding people with visual impairments (VI) through real-time video chat communication with sighted agents. We conducted a literature review and interviewed 12 RSA users to understand the technical and navigational challenges faced by both agents and users. The technical challenges were categorized into four groups: agents’ difficulties in orienting and localizing users, acquiring and interpreting users’ surroundings and obstacles, delivering information specific to user situations, and coping with poor network connections. We also presented 15 real-world navigational challenges, including 8 outdoor and 7 indoor scenarios. Given the spatial and visual nature of these challenges, we identified relevant computer vision problems that could potentially provide solutions. We then formulated 10 emerging problems that neither human agents nor computer vision can fully address alone. For each emerging problem, we discussed solutions grounded in human–AI collaboration. Additionally, with the advent of large language models (LLMs), we outlined how RSA can integrate with LLMs within a human–AI collaborative framework, envisioning the future of visual prosthetics.

Keywords: people with visual impairments; remote sighted assistance; conversational assistance; computer vision; artificial intelligence; human–AI collaboration; large language models (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2024
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/16/7/254/pdf (application/pdf)
https://www.mdpi.com/1999-5903/16/7/254/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:16:y:2024:i:7:p:254-:d:1437984

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().