Real-time decoding of question-and-answer speech dialogue using human cortical activity

David A. Moses, Matthew K. Leonard, Joseph G. Makin and Edward F. Chang ()
Additional contact information
David A. Moses: Department of Neurological Surgery and the Center for Integrative Neuroscience at UC San Francisco
Matthew K. Leonard: Department of Neurological Surgery and the Center for Integrative Neuroscience at UC San Francisco
Joseph G. Makin: Department of Neurological Surgery and the Center for Integrative Neuroscience at UC San Francisco
Edward F. Chang: Department of Neurological Surgery and the Center for Integrative Neuroscience at UC San Francisco

Nature Communications, 2019, vol. 10, issue 1, 1-14

Abstract: Abstract Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance’s identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate.

Date: 2019
References: Add references at CitEc
Citations: View citations in EconPapers (3)

Downloads: (external link)
https://www.nature.com/articles/s41467-019-10994-4 Abstract (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:natcom:v:10:y:2019:i:1:d:10.1038_s41467-019-10994-4

Ordering information: This journal article can be ordered from
https://www.nature.com/ncomms/

DOI: 10.1038/s41467-019-10994-4

Access Statistics for this article

Nature Communications is currently edited by Nathalie Le Bot, Enda Bergin and Fiona Gillespie

More articles in Nature Communications from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().