Multimodal brain-computer interface for robotic control: integration of real-time gaze tracking and EEG-based motor imagery.

Individuals with upper limb dysfunction face significant challenges in performing everyday tasks, often depending on healthcare professionals, caregivers, or family members. Such reliance places a continuous burden on helpers who must remain available for assistance. To address these challenges, this study investigated a virtual hybrid brain-computer interface (BCI) system that integrates gaze tracking with motor imagery (MI) to control a robotic arm, potentially reducing the dependency on human support. Twenty healthy, right-handed participants took part in a virtual game environment where they controlled a robotic arm using both gaze tracking and MI. During an initial training phase, participants' electroencephalography (EEG) signals were recorded with an EEG cap. These signals were then processed and classified using the common spatial pattern (CSP) algorithm and linear discriminant analysis (LDA). In parallel, a webcam was used for real-time gaze calibration to enable accurate target selection. In the subsequent testing phase, MI commands directed the virtual robot toward predetermined targets in a Unity-based game. Training accuracy consistently outperformed online testing accuracy. The MI signal classification achieved a true positive (TP) rate of approximately 75.5%, while a significant negative correlation (r = - 0.45) was observed between MI classification accuracy and game completion times, suggesting that higher MI accuracy led to quicker task execution. These findings demonstrate the potential of combining gaze tracking with MI-based BCI for robotic control as an assistive technology for upper limb impairments. Despite its promise, technical limitations indicate that further improvements are needed to enhance system robustness, practicality, and usability for everyday activities.