Hardware-accelerated integrated optoelectronic platform towards real-time high-resolution hyperspectral video understanding

Maksim Makarenko, Arturo Burguete-Lopez, Qizhou Wang, Silvio Giancola, Bernard Ghanem, Luca Passone and Andrea Fratalocchi ()
Additional contact information
Maksim Makarenko: King Abdullah University of Science and Technology
Arturo Burguete-Lopez: King Abdullah University of Science and Technology
Qizhou Wang: King Abdullah University of Science and Technology
Silvio Giancola: King Abdullah University of Science and Technology
Bernard Ghanem: King Abdullah University of Science and Technology
Luca Passone: King Abdullah University of Science and Technology Research Park Headquarters - Level 1 - Office 2225
Andrea Fratalocchi: King Abdullah University of Science and Technology

Nature Communications, 2024, vol. 15, issue 1, 1-12

Abstract: Abstract Recent advancements in artificial intelligence have significantly expanded capabilities in processing language and images. However, the challenge of comprehensively understanding video content still needs to be solved. The main problem is the requirement to process real-time multidimensional video information at data rates exceeding 1 Tb/s, a demand that current hardware technologies cannot meet. This work introduces a hardware-accelerated integrated optoelectronic platform specifically designed for the real-time analysis of multidimensional video. By leveraging optical information processing within artificial intelligence hardware and combining it with advanced machine vision networks, the platform achieves data processing speeds of 1.2 Tb/s. This capability supports the analysis of hundreds of frequency bands with megapixel spatial resolution at video frame rates, significantly outperforming existing technologies in speed by three to four orders of magnitude. The platform demonstrates effectiveness for AI-driven tasks, such as video semantic segmentation and object understanding, across indoor and aerial scenarios. By overcoming the current data processing speed limitations, the platform shows promise in real-time AI video understanding, with potential implications for enhancing human-machine interactions and advancing cognitive processing technologies.

Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41467-024-51406-6 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-51406-6

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-024-51406-6

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().