Using deep neural networks to disentangle visual and semantic information in human perception and memory

Adva Shoham (), Idan Daniel Grosbard, Or Patashnik, Daniel Cohen-Or and Galit Yovel ()
Additional contact information
Adva Shoham: Tel Aviv University
Idan Daniel Grosbard: Tel Aviv University
Or Patashnik: Tel Aviv University
Daniel Cohen-Or: Tel Aviv University
Galit Yovel: Tel Aviv University

Nature Human Behaviour, 2024, vol. 8, issue 4, 702-717

Abstract: Abstract Mental representations of familiar categories are composed of visual and semantic information. Disentangling the contributions of visual and semantic information in humans is challenging because they are intermixed in mental representations. Deep neural networks that are trained either on images or on text or by pairing images and text enable us now to disentangle human mental representations into their visual, visual–semantic and semantic components. Here we used these deep neural networks to uncover the content of human mental representations of familiar faces and objects when they are viewed or recalled from memory. The results show a larger visual than semantic contribution when images are viewed and a reversed pattern when they are recalled. We further reveal a previously unknown unique contribution of an integrated visual–semantic representation in both perception and memory. We propose a new framework in which visual and semantic information contribute independently and interactively to mental representations in perception and memory.

Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://www.nature.com/articles/s41562-024-01816-9 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:nathum:v:8:y:2024:i:4:d:10.1038_s41562-024-01816-9

Ordering information: This journal article can be ordered from
https://www.nature.com/nathumbehav/

DOI: 10.1038/s41562-024-01816-9

Access Statistics for this article

Nature Human Behaviour is currently edited by Stavroula Kousta

More articles in Nature Human Behaviour from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().