Deep Neural Network-Based Modeling of Multimodal Human–Computer Interaction in Aircraft Cockpits

Li Wang (), Heming Zhang and Changyuan Wang
Additional contact information
Li Wang: School of Electronic & Electrical Engineering, Baoji University of Arts and Sciences, Baoji 721016, China
Heming Zhang: School of Optoelectronic Engineering, Xi’an Technological University, Xi’an 710000, China
Changyuan Wang: School of Computer Science, Xi’an Technological University, Xi’an 710021, China

Future Internet, 2025, vol. 17, issue 3, 1-15

Abstract: Improving the performance of human–computer interaction systems is an essential indicator of aircraft intelligence. To address the limitations of single-modal interaction methods, a multimodal interaction model based on gaze and EEG target selection is proposed using deep learning technology. This model consists of two parts: target classification and intention recognition. The target classification model based on long short-term memory networks is established and trained by combining the eye movement information of the operator. The intention recognition model based on transformers is constructed and trained by combining the operator’s EEG information. In the application scenario of the aircraft radar page system, the highest accuracy of the target classification model is 98%. The intention recognition rate obtained by training the 32-channel EEG information in the intention recognition model is 98.5%, which is higher than other compared models. In addition, we validated the model on a simulated flight platform, and the experimental results show that the proposed multimodal interaction framework outperforms the single gaze interaction in terms of performance.

Keywords: deep learning; multi-modal; long short-term memory network; transformer; human–computer interaction (search for similar items in EconPapers)
JEL-codes: O3 (search for similar items in EconPapers)
Date: 2025
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/1999-5903/17/3/127/pdf (application/pdf)
https://www.mdpi.com/1999-5903/17/3/127/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jftint:v:17:y:2025:i:3:p:127-:d:1611056

Access Statistics for this article

Future Internet is currently edited by Ms. Grace You

More articles in Future Internet from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().