1660 results about "Eye movement" patented technology

The invention discloses an intelligent system and method for early screening of depression based on electroencephalogram-eye movement multi-modal data fusion. According to the system, in a virtual reality situation, emotion and cognitive response of a subject are stimulated through an elaborately-designed cognitive task, and brain electrical activity and sight behavior data are synchronously collected by adopting a wearable EEG device and a high-precision eye tracker. After signal preprocessing and feature extraction, deep fusion of multi-modal data is realized by using a weighted fusion formula or a graph neural network, the weight of each modal is automatically adjusted, interaction features between electroencephalogram and eye movement are extracted, and then the depression risk of a subject is accurately judged through a classifier. Experiments show that the method can effectively capture weak physiological abnormalities of mild depression patients, has the advantages of high sensitivity, high accuracy and real-time online screening, and meets the requirements of non-invasive, portable and intelligent clinical application.

The invention discloses a scene interactive AI rehabilitation evaluation training and health monitoring system, and relates to the technical field of rehabilitation medical treatment and artificial intelligence, a semantic perception module is used for collecting and recognizing voice input, facial expressions, action tracks and eye movement paths of a user in a training process, and extracting context parameters; the knowledge-driven training generation module is used for calling a rehabilitation knowledge graph constructed by a graph neural network based on context parameters and individual training history, and generating a multi-path training scheme; training a feedback regulation engine, collecting posture offset, physiological stress and emotion feedback, and dynamically adjusting task difficulty, rhythm and prompt mode based on a dual-channel reinforcement learning model; the prediction module fuses training and monitoring data, and predicts a network identification function degradation risk through degradation driving; the cloud edge fusion platform is used for realizing task quick response and graph strategy iterative updating; according to the invention, the individuation, self-adaption and intelligent prediction capabilities of rehabilitation training are improved, and the rehabilitation effect and the system practicability are obviously optimized.

The invention relates to a remote education data processing system which comprises the following steps: under a remote teaching task, pre-defining a task intention and a data expectation point; a semantic timestamp and a task binding label are printed on each data fragment; mapping the collected confusion data including silence, eye movement drift and prediction into a unified learning semantic vector; a micro-expression + interactive behavior + time sequence decision path ternary modeling mode is introduced, and a potential cognitive intention corresponding to the feature combination is recognized; teaching context information is fused; constructing a cognitive state mapping model; reasoning a current cognitive state label from multi-modal sensing data; searching intervention track VS effect feedback data in a historical database; generating a predicted intervention behavior sequence by using a sequence modeling algorithm; a dynamic combination suggestion chain including light prompt, content reconstruction, personalized practice and tutoring invitation is adopted; and superposing the cognitive state sequences of all students into a group cognitive trajectory map.

The invention provides an emotion recognition method based on electroencephalogram eye movement multi-mode cross-attention feature fusion, and the method comprises the steps: firstly carrying out the preprocessing of an emotion recognition public data set, and building a corresponding training set and a test set; secondly, constructing an electroencephalogram eye movement multi-mode cross-attention feature fusion emotion recognition model; then carrying out model training and performance testing; finally, an electroencephalogram eye movement multi-mode online emotion recognition system is built, and the effectiveness of the method is verified. The method has the advantages that an electroencephalogram eye movement multi-mode cross-attention feature fusion emotion recognition model is designed, a dynamic graph convolutional network and an attention mechanism are flexibly applied, meanwhile, multi-dimensional data are used, the problem that single-mode emotion recognition information is insufficient is solved, and the recognition accuracy is improved; and an electroencephalogram eye movement multi-mode online emotion recognition system is built, so that the interactivity is enhanced. The average recognition accuracy of the method reaches 97.94% and is superior to that of an existing optimal method, and meanwhile the accuracy of online emotion recognition reaches 87.4%.

The invention relates to the technical field of artificial intelligence, and discloses an English teaching training system and method fusing semantic matching and cognitive assessment, and the method comprises the steps: synchronously capturing a text response, a voice intonation, an eye movement track, a facial micro-expression and a touch rhythm generated in a learning process; semantic deviation deconstruction and cognitive intention quantization processing are carried out on the learning interaction original sequence, and a bidirectional deep semantic matching network is adopted to carry out context alignment on student answers and target corpora; based on the word meaning divergence point set and the cognitive load multi-scale vector, extracting a nonlinear diffusion trajectory of a learning state by using a time gating multi-layer recursive trajectory evolution algorithm; the knowledge point nodes, the deviation type nodes and the emotion triggering nodes associated with the emotion instability candidate segments are fused to construct a local learning map; and forming an emotion cognition feedback result driven by learning interest based on the local learning map and the self-adaptive error correction intervention sequence. The method has the advantage of improving the learning interest of students.

The invention relates to the field of man-machine interaction training, virtual reality and intelligent evaluation, and particularly discloses a psychological exercise ability training method and system based on mixed reality and behavior analysis. The method comprises the steps that a multi-scene psychological training environment is constructed and presented in mixed reality equipment, action behavior data, eye movement path data and operation response time data of a pilot in the training process are collected, multi-modal original behavior characteristics are constructed, and then a psychological exercise ability label set is constructed; and realizing capability-task modeling through matching of a capability label and a task structure, generating a personalized training path and a capability difference atlas based on difference analysis with a preset capability standard, adjusting task parameter configuration data, dynamically updating a training scene, generating a capability evolution trajectory, and outputting a training analysis result. In this way, accurate ability modeling, task dynamic adaptation and visual tracking of the training effect can be achieved, and the individuation and intelligence level of psychological exercise ability training is remarkably improved.

The invention discloses a multi-modal fusion alertness assessment method and system based on cross attention, and the method comprises the steps: carrying out the preprocessing of an original electroencephalogram signal of a user, and extracting a time-frequency domain-space feature # imgabs0 #; using a depth separable convolution module and an LSTM module to extract an electroencephalogram modal time dependency relationship feature # imgabs2 from the time-frequency domain-space feature # imgabs1 # in sequence; the method comprises the following steps: preprocessing an original electro-oculogram signal of a user to extract an eye movement feature # imgabs3 #, and obtaining an eye movement modal internal relation feature # imgabs5 # from the eye movement feature # imgabs4 # by using an outer product module and a CNN module in sequence; and performing multi-modal fusion on the electroencephalogram modal time dependency relationship characteristic # imgabs6 # and the eye movement modal internal relationship characteristic # imgabs7 # based on cross attention to obtain a fusion characteristic # imgabs8 #, and performing regression to obtain an alertness prediction result # imgabs9 # of the user. According to the method, intra-modal feature extraction and multi-modal feature fusion are respectively carried out on different information sources, so that the robustness and generalization of the features are improved, and the accuracy of alertness assessment is improved.

The invention provides an autism assessment system and method based on multi-modal time sequence data fusion, and relates to the technical field of children autism spectrum disorder assessment calculation processing. The invention innovatively provides a dual time sequence alignment method based on dynamic time warping (DTW) and LSTM prediction, the problem of time sequence dislocation of cross-modal data (heart rate / eye movement / limb movement) caused by acquisition frequency difference is solved, millisecond-level synchronization precision is realized, the technical problem of cross-modal data time sequence dislocation in an existing autism assessment system is solved, and the accuracy of time sequence alignment of the cross-modal data in the autism assessment system is improved. The small-range time deviation caused by acquisition equipment delay or physiological response difference is eliminated, and the robustness and reliability of the autism evaluation system are improved.

The invention discloses a multi-mode electroencephalogram analysis model construction method and an online processing method and system. According to the method, multi-modal physiological signals such as electroencephalogram, electrocardio, skin electrical activity and eye movement are collected, and an integrated multi-modal signal collection device is used for preprocessing and feature extraction. A modal expert system based on an adaptive Transform architecture is adopted to replace a standard feed-forward network so as to enhance the feature processing capability. By means of the method, the features can be complemented under the condition of mode or data missing, the robustness, real-time performance and accuracy of the brain-computer interface technology in multi-mode data processing are improved, and the method is particularly suitable for the fields of emotion recognition, cognitive load monitoring, neural rehabilitation and the like.

The invention provides a model-based interaction method and system, wearable equipment and a storage medium, and belongs to the technical field of intelligent interaction.The method comprises the steps that in response to a received interaction instruction, voice data, a gesture image, eye movement data and an environment image are obtained based on the interaction instruction; extracting user intention features based on the voice data, the gesture image and the eye movement data, and determining scene type features based on the environment image; determining an interaction theme based on the interaction instruction, obtaining user historical interaction information associated with the interaction theme from a context memory database, and generating a context feature vector based on the user historical interaction information; and inputting the user intention feature, the scene type feature and the context feature vector into an intention recognition model based on quantum enhancement to obtain a user intention, and generating interaction response data based on the user intention. According to the invention, the accuracy of user intention recognition can be improved, and the intelligence of interaction is improved.

The invention belongs to the technical field of brain-computer interfaces, and particularly relates to a brain-computer interface driven emotion recognition and self-adaptive regulation and control system which comprises a multichannel nerve-peripheral coupling module, an emotion intensity probability mapping module and a closed-loop nerve regulation and control current module. The multi-channel nerve-peripheral coupling module is used for realizing overall quantification of central and peripheral emotional physiology; the emotion intensity probability mapping module is used for generating continuous emotion probabilities ranging from 0 to 1 through normalization and nonlinear mapping by utilizing emotion energy and combining eye movement fatigue and electroencephalogram entropy; and the closed-loop nerve regulation and control current module is used for dynamically adjusting the transcranial electrical stimulation intensity within the safety current upper limit according to the difference value between the emotion probability and the expected target. According to the invention, the recognition precision, the response speed and the use comfort are obviously improved.

The invention belongs to the technical field of intelligent education, and particularly relates to a classroom attention detection method and system based on multi-modal data fusion. Aiming at the problems of high equipment cost, low multi-source data fusion efficiency, insufficient privacy protection and the like in the prior art, the invention provides the following solutions: collecting face, eye movement, posture, voice signals and heart rate variability data of a student through a sensor; multi-modal data synchronization is realized by adopting a time sequence alignment algorithm; respectively extracting a visual attention feature, a voiceprint matching feature and a physiological wake-up feature by using a lightweight deep learning model; constructing a multi-modal data fusion network, and dynamically adjusting a feature weight in combination with a classroom scene; attention anomaly detection is realized by adopting a hybrid model, and real-time early warning is output through edge computing equipment. The method has the beneficial effects that the hardware cost is greatly reduced while the detection precision is ensured, and the privacy of students is effectively protected; a dynamic weight distribution mechanism improves the adaptability of different teaching scenes.

The invention discloses an emotional disorder detection method based on a multi-modal heterogeneous graph convolutional neural network, and the method comprises the steps: collecting the electroencephalogram, expression pictures, eye movement data and gait video frame data of a subject, and aligning the data through an asynchronous time sequence cross-modal alignment and group horizontal alignment method. Thirdly, decomposing data of each mode by utilizing low-rank matrix decomposition of multi-mode characteristics, and performing alignment of different tasks with a nerve activation path; carrying out heterogeneous graph convolution channel path coding on each channel and embedding a modal; and finally carrying out lightweight cross-modal fusion on the obtained data stream. The invention further provides an emotional disorder detection device. According to the method provided by the invention, the recognition of the type of the emotional disorder is realized through the acquired multi-modal data including electroencephalogram, eye movement, micro-expression and gait information.

The invention discloses an automatic tracking system based on XR space, and relates to the technical field of XR space, the system comprises a sight tracking module, a scene pairing module, an interactive navigation module and a feedback adjustment module; the technical key points are as follows: establishing a preset instruction library, accurately mapping recognized eyeball poses to a specific analysis instruction in an XR space, and obtaining a fixation area and a movement direction in the XR space; according to the method, rough and fine virtual scene division is carried out based on eye movement data, the virtual scene where the user is located is judged in real time, the immersion and participation sense of the user are greatly improved, then an initial rotation value and guide time are set, a preliminary calculation model is dynamically adjusted according to user feedback, guide simulation is optimized, and it is ensured that the user obtains vivid experience.

The invention relates to the field of electroencephalogram signal processing, and particularly discloses a multi-mode electroencephalogram characteristic cognitive ability evaluation system based on a neural network, comprising: an acquisition module used for synchronously acquiring whole brain signals and eye movement data during pilot simulation training; the data preprocessing module is used for primarily processing the received whole brain signals and the eye movement data, monitoring the primarily processed data in real time by utilizing a preset monitoring mechanism, and triggering to generate interaction test starting signals if preset symbolic fluctuation occurs in the whole brain signals or the eye movement data; and the test interaction sub-module is used for generating arithmetic test questions according to a preset rule after receiving the interaction test starting signal, and synchronously recording answer correlation parameters of the pilot. By adopting the technical scheme of the invention, the deep-level reaction mechanism of the brain under the stimulation of the special complex cognitive task of flight training can be fully considered, and the cognitive load change of the pilot in the simulated training can be comprehensively and accurately reflected.

The invention relates to the technical field of artificial intelligence and biomedical sensing, and discloses a cognitive disorder risk identification device and method integrating multi-modal physiological data, and the device comprises an EEG module, an eye movement module, an fNIRS module, an edge calculation module and an acceleration sensor; the EEG module collects EEG data, the eye movement module collects eye movement eye-tracing data, and the fNIRS module collects near infrared spectrum fNIRS data and inputs the data to the edge calculation module; the edge calculation module runs the multi-modal space-time attention fusion model to output a cognitive impairment risk assessment result by using the multi-modal space-time attention fusion model; the acceleration sensor operates based on an operation dynamic filtering algorithm of a motion accelerometer to inhibit signal drift caused by head motion. According to the invention, through the modularized head-mounted multi-mode edge device, the physiological data is collected and edge end processing and cognitive disorder recognition and screening are carried out in combination with the edge calculation module, so that early recognition and screening of neurodegenerative diseases can be conveniently and effectively carried out at low cost.

The invention discloses a pilot manipulation ability evaluation method and system based on multi-modal data and online learning driving. The method comprises the steps of determining an input parameter set, establishing a flight vision time sequence fusion network, and extracting each feature of the parameter set. And completing feature fusion of the multi-modal data through an attention mechanism and global aggregation. And constructing a pilot maneuvering ability evaluation architecture, and generating an offline pilot maneuvering ability evaluation model by designing a loss function based on a comprehensive scoring error and sub-scoring consistency constraint. Flight data and eye movement data streams generated in real time are fragmented and preprocessed by using a sliding window mechanism, a lightweight model based on pruning and parameter quantification technologies is designed for real-time scoring requirements, and feature expression ability in an offline pilot manipulation ability evaluation model is migrated to a lightweight online evaluation model. The method can be used for improving a pilot training scheme, optimizing the man-machine interface design of a cockpit, and improving the efficiency and safety of flight operation.

The invention discloses a dangerous driving critical state identification method, and relates to the technical field of intelligent driving safety. According to the method, multi-mode signals of eye movement, electrocardio, skin electricity, vehicle operation and the like are collected and converted into a unified phase field, and the synchronous coherence of the unified phase field is analyzed; the individual phase dynamics manifold of the driver is learned on line by using a Shenchang differential equation, and a system instability precursor is identified by detecting the behavior that a state point escapes from a steady state attractor; further, multi-dimensional indexes such as synchronous collapse and topological fracture are fused, collapse time is estimated in combination with a Lyapunov index, and an advanced early warning instruction is generated; and finally, based on the model predictive control and the personalized phase response curve, generating and executing targeted multi-mode phase reset intervention, and forming a sensing-early warning-intervention active safety closed loop. According to the invention, normal form transformation from post-event alarm to beforehand regulation and control is realized, and early warning advancement and intervention accuracy are improved.

The invention provides a multifunctional inspection device detection system based on a distribution network mobile operation terminal, and relates to the technical field of electric power detection, and the multifunctional inspection device detection system comprises a portable main terminal of a dynamic collaborative architecture, a wearable terminal for augmented reality interaction and a distributed heterogeneous sensing cluster, which form a closed-loop data chain through an electric power dedicated low-delay wireless communication network; the portable main terminal is integrated with a heterogeneous computing unit with adaptive computing power distribution, can perform real-time fusion analysis on multi-dimensional sensing data, and outputs a visual result containing fault location and confidence; the wearable terminal superposes fault information to a real scene in a three-dimensional marking form through a virtual-real fusion positioning technology, and supports eye movement and voice collaborative interaction; multi-module collaborative acquisition of the distributed heterogeneous sensing cluster is combined with a nanosecond timestamp synchronization mechanism, so that the limitation of traditional single parameter detection is broken through, rich and synchronous basic data is provided for subsequent analysis, and the comprehensiveness of routing inspection is greatly improved.

A user workload state processing method and apparatus based on multimodal data, and a device. The method comprises: acquiring state information of a first user; determining workload information of the first user on the basis of the acquired state information, wherein the state information comprises at least two of the following information: physiological information, eye movement information, electroencephalogram information, brain function imaging information, motion capture information, spatiotemporal acquisition information, behavior acquisition information, and facial expression and state information; and feeding back the workload information to a first management account, and formulating a training scheme matched with the workload information. According to the embodiments of the present application, workload of users can be identified on the basis of multimodal data of users.

The invention provides a neural rehabilitation training method and system integrating a brain-computer interface and virtual reality, and relates to the technical field of brain-computer interfaces. The method comprises the following steps: constructing an aligned multi-modal feature sequence by collecting electroencephalogram, myoelectricity, joint kinematics, eye movement and physiological load signals; generating an immersion parameter prescription in the baseline stage and setting a time delay and synchronization strategy; according to the nerve quality index, performing cooperative self-adaption of decoder parameters, prescriptions and peripheral assistance; establishing a drift model after the session to update the prior and shorten the re-calibration time; and monitoring dizziness and task load in real time and executing grading treatment. According to the invention, stable closed-loop individualized rehabilitation training is realized, the decoding performance and the rehabilitation effect are improved, and the safety and long-term convergence are ensured.

Disclosed herein are system, method, and device embodiments for determining the mental state of a driver using gaze tracking and object detection. Specifically, an Advanced Driver Assistance System may determine if the driver's gaze is associated with spatial locations of objects of interest in the surrounding environment. The surrounding environment may be represented with a 360-degree, three-dimensional model generated by the external-facing cameras and other sensors. The driver's gaze direction may be represented by a heatmap derived based on the driver's eye movements. Whether or not the driver's gaze is associated with a spatial location of an object may be used as evidence about the driver's mental state. The system may perform appropriate actions based on the mental state including generating alerts, engaging mechanical systems, and adjusting parameters in ancillary safety systems to increase or decrease tolerance. The mental state may be tracked throughout a driving session and stored in a driver profile.

A method to prevent intoxicated operation is described. The method includes providing instructions to a vehicle user to position a vehicle user body portion posture in a predefined alignment. The method further includes obtaining the vehicle user body portion posture from a first vehicle detector, and determining whether the vehicle user body portion posture is in the predefined alignment. The method includes activating a plurality of vehicle visual indicators to illuminate in a predefined manner when the vehicle user body portion posture is in the predefined alignment. The method further includes providing instructions to the vehicle user to move vehicle user eyes to track the plurality of vehicle visual indicators, and obtaining a vehicle user eye movement from a second vehicle detector. The method further includes determining whether the vehicle user eye movement meets a predetermined condition, and actuating a control action accordingly.

The invention discloses a multi-mode emotion continuous recognition method for medical treatment, belongs to the technical field of artificial intelligence and medical treatment information, and mainly aims to simulate the dynamic change process of emotion by establishing a Neural ODEs framework and overcome the static property and discreteness of emotion modeling in a traditional method. Through a causal inference technology, emotional features are separated from individual-independent physiological differences, and the generalization ability across individuals is improved. A self-supervised learning method is utilized, the synergistic effect between the EEG and the eye movement signal is improved through cross-modal contrast learning, and the emotion recognition precision is enhanced. The calculation complexity is reduced through a dynamic sparse attention mechanism, and meanwhile, focusing is performed on a key time slice in emotion recognition. Through multi-task joint learning, the model learns multiple tasks such as emotion intensity regression and tested identity recognition during emotion classification, and the personalized emotion recognition capability is improved.

The invention discloses a data labeling method and system based on user behaviors and attention tracking, and the method comprises the steps: synchronously collecting multi-source behavior signals of a mouse, a keyboard, eye movement and the like of a doctor in real time, combining identity and interface metadata, and carrying out the standardized normalization, abnormality elimination and short time sequence behavior unit division. And extracting individual behavior micro-modes by using unsupervised clustering, and constructing a behavior portrait library. Through multi-modal time sequence modeling and a self-adaptive space-time attention mechanism, behavior characteristics, an interface area and a report text are deeply fused, a multi-level correlation probability is output, and high-precision automatic tagging of content and an image area is realized.

The invention discloses a high-precision multi-modal twin model assembly method based on a large language model and a multiple docking optimization mechanism. The whole system framework is composed of three modules: a database, an interaction system and human factors. Wherein the database module integrates a model library module, a user interaction data recording module and a temporary data recording module, and is used for storing and managing various data information required for constructing a virtual scene; the interactive system module is used as a core part, integrates a model dynamic loading module, a model accurate positioning module and a model adaptive assembly module, realizes instruction analysis and model generation by relying on a large language model, and combines a multi-docking optimization mechanism of eye movement tracking, gesture recognition, automatic docking and logic rule binding to realize multi-docking. Efficient and accurate interaction between the user and the database is effectively guaranteed; the method not only provides a solid foundation for virtual experiments and virtual-real symbiosis, but also significantly improves the construction efficiency and precision of digital twin model assembly.

The invention provides a low-altitude typical scene-oriented aircraft cockpit simulation method, system and device and a storage medium, and the method comprises the steps: constructing a dynamic extensible low-altitude scene library through fusing mixed reality (MR) and a high-precision digital twin model; in combination with a man-machine in-the-loop multimode interaction technology (eye movement-gesture-voice cooperation), the operation naturalness is improved; and an intelligent optimization algorithm driven by reinforcement learning is introduced, adaptive adjustment of flight control parameters is realized, and finally a'perception-decision-evaluation 'closed-loop training system is formed. According to the invention, digital twinning, mixed reality (MR), multi-modal sensing and adaptive control technologies are fused, pilot low-altitude flight skill training, man-machine work efficiency evaluation and intelligent aircraft control system verification are carried out, and the method can be expanded to UAM operator training and unmanned aircraft simulation testing. The technical barrier of low-altitude flight skill training at present is solved, and the technical blank of low-altitude dynamic scene simulation is filled up.

The invention discloses an aircraft control simulation platform construction method, and the method comprises the steps: constructing a terrain multi-dimensional model, fusing a cross-scale meteorological particle system, and dynamically obtaining an environment disturbance factor; a space obstacle dynamic database is generated in combination with millimeter-level space positioning data, and virtual collision detection is carried out; performing eye movement-gesture collaborative decision, and identifying a control instruction with high confidence; the six-degree-of-freedom kinetic equation is coupled with an unreal engine interface in real time, and a high-fidelity instrument is driven to feed back; generating a training defect thermodynamic diagram, triggering an adaptive optimization strategy and iterating platform parameters; the invention provides a solution integrating multi-modal high-precision data acquisition, dynamic environment-physical rule coupling, intelligent decision and closed-loop optimization so as to realize aircraft control simulation training with high immersion degree, high fidelity and high adaptability.

The invention provides a visual fatigue relieving method based on ambient light self-adaption and an AI algorithm, and relates to the technical field of visual health protection. The visual fatigue relieving method based on ambient light self-adaption and the AI algorithm comprises the following specific steps: S1, data acquisition: acquiring a user eye image sequence in real time through a high-frame-rate camera; and S2, parameter extraction: when the user uses the eye for the first time, testing the eyes by combining the adjustable light source with ambient light. Multi-dimensional features (such as eyeball movement, blinking mode, pupil function and the like) of eyes are collected in real time through a high-frame-rate camera, an individualized base line is established and a fatigue rate value is dynamically calculated in combination with data of an ambient light sensor, and conversion from passive response to active prevention is realized. Through a hierarchical intervention strategy (such as brightness adjustment, blue light control and forced rest), the visual load is remarkably reduced, the intervention efficiency is improved, and the problem of insufficient adjustment hysteresis and individual adaptability in the prior art is solved.