Brain-Computer Interfaces and HCI: The Tech Path from Eye Tracking to Neural Communication

1. From Motor Impairment to Multimodal Interaction: Real-World HCI Needs

Patients affected by neurodegenerative diseases often gradually lose both motor skills and the ability to speak. In such cases, traditional input methods like keyboards, mice, and touchscreens can no longer meet basic communication needs. To enable these users to express intent, control devices, and participate in society, HCI technology must shift toward richer physiological and neural signal channels.

Eye tracking is currently a widely applied assistive interaction solution. The basic approach involves capturing eye images using infrared light sources or cameras, estimating the user’s gaze direction based on the relative position changes between the pupil center and the corneal reflection point, and mapping the gaze point to screen coordinates to execute actions like selection, clicking, and typing. For users who retain some eye movement capability, eye tracking provides stable interaction without the need for physical contact.

However, eye tracking has distinct limitations. Prolonged use easily leads to visual fatigue. Factors such as ambient light, glasses reflection, blinking, and nystagmus can affect accuracy. As the disease progresses, extraocular muscle function may decline, causing the eye movement signal to weaken or disappear entirely. Consequently, the industry is exploring more fundamental interaction methods—directly extracting control intent from neural activity—bringing brain-computer interfaces (BCI) into clinical and engineering focus.

2. Engineering Implementation and Key Constraints of Eye Tracking

A typical eye tracking system consists of image acquisition, feature extraction, gaze estimation, interaction mapping, and feedback execution. The image acquisition phase utilizes near-infrared cameras to obtain clear eye images even in low-light environments. During feature extraction, the system locates the pupil boundary and corneal reflection point, calculating the gaze direction based on their relative offset.

In engineering practice, gaze estimation accuracy is influenced by multiple factors. First, individual differences—such as varying pupil shapes, eyelid occlusion, and corneal curvature—require personalized calibration. Second, head pose changes: slight head movements alter the geometric relationship between the camera and the eye, necessitating head pose compensation. Third, real-time requirements: noticeable latency in updating gaze coordinates causes operational lag, degrading the continuous input experience.

To minimize error, some solutions fuse multiple features, such as utilizing binocular information, applying historical trajectory smoothing, and predicting user intent based on UI elements. For instance, when a user gazes at a specific button area, the system can snap to and confirm the button’s hotzone, even if the raw gaze point jitters. This improves click accuracy to a certain extent.

Despite these optimizations, eye tracking is not the ultimate solution. For users with continuously degrading eye movement capabilities, the system must consider multi-channel backup and migration strategies. A common approach is to preserve low-load input channels like voice, minor electromyography (EMG), or switch signals, while proactively evaluating whether the user is suitable for BCI training.

3. Brain-Computer Interfaces: From Neural Signals to Actionable Commands

The goal of a BCI is to bypass the peripheral nervous system and muscles, establishing a direct information pathway between the brain and external devices. Based on signal acquisition methods, BCIs are generally categorized into invasive, semi-invasive, and non-invasive types. Non-invasive solutions use scalp electrodes to record electroencephalogram (EEG) signals, offering high safety but limited signal resolution. Invasive solutions implant electrodes intracranially to acquire higher-quality neural signals but involve risks like surgery, infection, and long-term stability issues. Semi-invasive solutions fall somewhere in between.

From an engineering pipeline perspective, a BCI system includes signal acquisition, preprocessing, feature extraction, pattern recognition, command mapping, and device execution. Acquired raw signals often contain noise such as electrooculography (EOG), EMG, and power frequency interference, requiring filtering, artifact removal, and segmentation. Subsequently, the system extracts features related to motor imagery, visual attention, or specific cognitive tasks, such as energy changes in specific frequency bands, event-related potentials (ERP), or slow cortical potentials (SCP).

During the training phase, users follow prompts to imagine movements or focus on specific stimuli while the system builds a classification model based on synchronously recorded data. Classification results can be mapped to discrete commands like “left,” “right,” “confirm,” or “back,” or used to control cursor movement, character selection, or robotic arm motion. Under current technological conditions, there is still significant room for improvement in the instruction transfer rate, accuracy, and stability of BCIs. However, in certain scenarios, they already provide meaningful communication capabilities.

For user groups with conditions like Amyotrophic Lateral Sclerosis (ALS), the long-term use of BCIs presents unique challenges. Patients may struggle to maintain attention for extended periods; electromagnetic interference and changes in electrode contact within home environments affect signal quality; and caregivers require simple, reliable operational workflows. Therefore, BCIs must evolve beyond laboratory demonstrations toward low-maintenance costs, adaptive calibration, and remote monitoring.

4. Clinical Research and Real-World Scenario Validation

Advancing BCIs from the lab to practical use relies heavily on clinical research. Unlike standard software development, HCI systems for patients with neurological diseases must first meet safety, efficacy, and ethical compliance requirements. Research institutions must establish collaborative mechanisms with hospitals and rehabilitation centers, embedding technical validation into actual clinical workflows.

Clinical research typically begins with screening and assessment, covering the user’s state of consciousness, cognitive ability, visual or auditory function, and residual motor skills. The research team then equips the user with the device, completing baseline calibration and training tasks. Evaluation metrics focus not only on offline classification accuracy but also on online task completion time, error rate, subjective user load, and caregiver operation difficulty.

Notably, real-world environments are far more complex than laboratories. For example, a hospital ward may contain multiple monitoring devices generating electromagnetic noise across various frequency bands. Factors like phlegm, body position changes, and emotional fluctuations can affect signal stability. State variations across different times of day can also lead to model drift. Therefore, systems must possess capabilities like automatic signal quality detection, dynamic model weight updating, and abnormal state alerting, rather than relying on a one-time training result for long-term use.

Furthermore, clinical research involves data privacy, informed consent, result interpretation, and withdrawal mechanisms. Technology trials involving high-risk populations must fully respect the user’s right to autonomous decision-making. Especially concerning invasive solutions, long-term implantation risks must be carefully evaluated and thoroughly compared against non-invasive, eye tracking, and EMG alternatives.

5. From Eye Tracking to BCI: The Evolution Logic of Multimodal Fusion

From an engineering evolution perspective, eye tracking and BCI are not mutually exclusive alternatives; rather, they are better integrated into a unified multimodal interaction architecture. Different channels have unique advantages: eye tracking offers high speed and low learning costs when eye movement is functional; BCI can provide basic control capabilities even when eye movement is restricted; voice, EMG, and switch signals serve as low-load supplements.

The goal of multimodal fusion is not simple addition, but the dynamic selection of the most appropriate input channel based on user state, task type, and environmental conditions. The system can establish a channel health evaluation mechanism, monitoring metrics like eye movement signal stability, EEG signal-to-noise ratio, and voice recognition confidence. When a channel’s quality degrades, the system automatically elevates the weight of other channels or even switches the primary input method.

In terms of software architecture, such systems typically adopt a layered design: “Signal Acquisition Layer — Feature Processing Layer — Intent Understanding Layer — Application Control Layer.” The Signal Acquisition Layer connects various sensor devices. The Feature Processing Layer handles filtering, denoising, feature extraction, and quality assessment. The Intent Understanding Layer fuses multi-channel features into a unified interaction intent. The Application Control Layer translates this intent into specific device commands. Through standardized data interfaces and message formats, the complexity of integrating new input channels is significantly reduced.

For developers, designing such systems also requires attention to latency budgets, false trigger rates, and recoverability. In assistive interaction scenarios, a false trigger can lead to severe consequences. Therefore, critical operations must incorporate secondary confirmation, undo mechanisms, or safety boundaries. Concurrently, the system should log complete interaction records to facilitate failure analysis and continuous model optimization.

6. Future Direction: From Device Control to Thought-Based Communication

As the quality of neural signal acquisition, decoding algorithms, and computational platform capabilities improve, BCIs are evolving from simple discrete command control to continuous motion control and more natural semantic interaction. In the future, if language-related neural activity can be decoded more accurately, users may no longer need to select characters one by one, but instead output content in a manner much closer to natural expression.

This involves several core technical challenges. First, the complexity of neural representation: language production and comprehension involve the coordination of multiple brain regions, making it difficult to capture comprehensively with a small number of electrodes. Second, temporal alignment: there are complex timing relationships between mental activity, external events, and user intent, requiring high-quality annotated data for training. Third, individual differences and cross-day migration: the same model may perform significantly differently across different users and days, necessitating robust transfer learning and adaptive mechanisms.

Another crucial direction is closed-loop feedback. A one-way output BCI only converts neural signals into commands. A closed-loop system, however, feeds the results back to the user, creating an “intent — execution — feedback — adjustment” loop. For instance, while a user looks at a virtual keyboard, the system determines the target character based on EEG and eye tracking signals, providing instant feedback via highlighting or sound. This closed-loop mechanism helps users learn to regulate their own neural activity and assists the system in continuously refining its model.

At the engineering resource level, BCIs demand high computational power, storage, and low-latency transmission. Especially in home environments, edge devices must handle signal preprocessing and preliminary inference, while the cloud assumes responsibilities like model training, large model inference, and remote operations. Through “device-edge-cloud” collaboration, real-time performance is ensured while personalized models are continuously optimized.

7. General Design Principles for Engineering Implementation

Whether utilizing eye tracking, BCIs, or multimodal fusion systems, the ultimate goal is translating these into maintainable, deployable, and scalable engineering solutions. The following universal design principles apply to most assistive interaction systems.

First, center channel selection on user state. The system should not force users into a single input method, but rather provide a smoothly migratable path based on changes in residual abilities. For example, prioritize eye tracking while eye movement is viable, automatically increase EEG signal weight as eye tracking quality declines, and enable voice or switch inputs when necessary.

Second, treat personalized modeling as a default capability. Physiological signals vary greatly among individuals, meaning models cannot rely solely on public datasets. The system should support short-duration user calibration, online incremental learning, and periodic retraining to continuously adapt to individual state changes.

Third, prioritize anomaly detection and degradation strategies. When signal quality suddenly drops, model confidence is insufficient, or device connections fail, the system should enter a safe state—such as pausing execution, prompting recalibration, or switching to a backup channel—rather than continuing to output unreliable commands.

Fourth, maintain restraint in interaction design. In assistive contexts, complex interfaces, multi-level menus, and frequent confirmations increase cognitive load. Developers should prioritize flat structures, predictable layouts, and clear feedback, minimizing the number of steps required for a user to complete a task.

8. Conclusion

From eye tracking to brain-computer interfaces, human-computer interaction technology is extending toward deeper physiological and neural signals. For users who have lost traditional input capabilities due to disease, this is not merely a technological evolution, but a means to regain the ability to express, communicate, and control their environment. In engineering, relying on a single channel is insufficient to cover all scenarios; multimodal fusion, adaptive modeling, clinical validation, and cloud-edge collaboration will become key pillars. In the future, as neural decoding accuracy and system stability improve, the goal of moving from “device control” to “thought-based communication” requires long-term exploration, but the technological path is becoming clearer. For developers and researchers, the current priority is validating every component in real-world scenarios, ensuring the technology transitions from a demonstration to a usable, reliable, and sustainable tool.