1117 results about "Eye movement" patented technology

The invention provides a model-based interaction method and system, wearable equipment and a storage medium, and belongs to the technical field of intelligent interaction.The method comprises the steps that in response to a received interaction instruction, voice data, a gesture image, eye movement data and an environment image are obtained based on the interaction instruction; extracting user intention features based on the voice data, the gesture image and the eye movement data, and determining scene type features based on the environment image; determining an interaction theme based on the interaction instruction, obtaining user historical interaction information associated with the interaction theme from a context memory database, and generating a context feature vector based on the user historical interaction information; and inputting the user intention feature, the scene type feature and the context feature vector into an intention recognition model based on quantum enhancement to obtain a user intention, and generating interaction response data based on the user intention. According to the invention, the accuracy of user intention recognition can be improved, and the intelligence of interaction is improved.

The invention belongs to the technical field of brain-computer interfaces, and particularly relates to a brain-computer interface driven emotion recognition and self-adaptive regulation and control system which comprises a multichannel nerve-peripheral coupling module, an emotion intensity probability mapping module and a closed-loop nerve regulation and control current module. The multi-channel nerve-peripheral coupling module is used for realizing overall quantification of central and peripheral emotional physiology; the emotion intensity probability mapping module is used for generating continuous emotion probabilities ranging from 0 to 1 through normalization and nonlinear mapping by utilizing emotion energy and combining eye movement fatigue and electroencephalogram entropy; and the closed-loop nerve regulation and control current module is used for dynamically adjusting the transcranial electrical stimulation intensity within the safety current upper limit according to the difference value between the emotion probability and the expected target. According to the invention, the recognition precision, the response speed and the use comfort are obviously improved.

The invention discloses a dangerous driving critical state identification method, and relates to the technical field of intelligent driving safety. According to the method, multi-mode signals of eye movement, electrocardio, skin electricity, vehicle operation and the like are collected and converted into a unified phase field, and the synchronous coherence of the unified phase field is analyzed; the individual phase dynamics manifold of the driver is learned on line by using a Shenchang differential equation, and a system instability precursor is identified by detecting the behavior that a state point escapes from a steady state attractor; further, multi-dimensional indexes such as synchronous collapse and topological fracture are fused, collapse time is estimated in combination with a Lyapunov index, and an advanced early warning instruction is generated; and finally, based on the model predictive control and the personalized phase response curve, generating and executing targeted multi-mode phase reset intervention, and forming a sensing-early warning-intervention active safety closed loop. According to the invention, normal form transformation from post-event alarm to beforehand regulation and control is realized, and early warning advancement and intervention accuracy are improved.

The invention provides a multifunctional inspection device detection system based on a distribution network mobile operation terminal, and relates to the technical field of electric power detection, and the multifunctional inspection device detection system comprises a portable main terminal of a dynamic collaborative architecture, a wearable terminal for augmented reality interaction and a distributed heterogeneous sensing cluster, which form a closed-loop data chain through an electric power dedicated low-delay wireless communication network; the portable main terminal is integrated with a heterogeneous computing unit with adaptive computing power distribution, can perform real-time fusion analysis on multi-dimensional sensing data, and outputs a visual result containing fault location and confidence; the wearable terminal superposes fault information to a real scene in a three-dimensional marking form through a virtual-real fusion positioning technology, and supports eye movement and voice collaborative interaction; multi-module collaborative acquisition of the distributed heterogeneous sensing cluster is combined with a nanosecond timestamp synchronization mechanism, so that the limitation of traditional single parameter detection is broken through, rich and synchronous basic data is provided for subsequent analysis, and the comprehensiveness of routing inspection is greatly improved.

The invention provides a neural rehabilitation training method and system integrating a brain-computer interface and virtual reality, and relates to the technical field of brain-computer interfaces. The method comprises the following steps: constructing an aligned multi-modal feature sequence by collecting electroencephalogram, myoelectricity, joint kinematics, eye movement and physiological load signals; generating an immersion parameter prescription in the baseline stage and setting a time delay and synchronization strategy; according to the nerve quality index, performing cooperative self-adaption of decoder parameters, prescriptions and peripheral assistance; establishing a drift model after the session to update the prior and shorten the re-calibration time; and monitoring dizziness and task load in real time and executing grading treatment. According to the invention, stable closed-loop individualized rehabilitation training is realized, the decoding performance and the rehabilitation effect are improved, and the safety and long-term convergence are ensured.

The invention discloses a multi-mode emotion continuous recognition method for medical treatment, belongs to the technical field of artificial intelligence and medical treatment information, and mainly aims to simulate the dynamic change process of emotion by establishing a Neural ODEs framework and overcome the static property and discreteness of emotion modeling in a traditional method. Through a causal inference technology, emotional features are separated from individual-independent physiological differences, and the generalization ability across individuals is improved. A self-supervised learning method is utilized, the synergistic effect between the EEG and the eye movement signal is improved through cross-modal contrast learning, and the emotion recognition precision is enhanced. The calculation complexity is reduced through a dynamic sparse attention mechanism, and meanwhile, focusing is performed on a key time slice in emotion recognition. Through multi-task joint learning, the model learns multiple tasks such as emotion intensity regression and tested identity recognition during emotion classification, and the personalized emotion recognition capability is improved.

The invention discloses a data labeling method and system based on user behaviors and attention tracking, and the method comprises the steps: synchronously collecting multi-source behavior signals of a mouse, a keyboard, eye movement and the like of a doctor in real time, combining identity and interface metadata, and carrying out the standardized normalization, abnormality elimination and short time sequence behavior unit division. And extracting individual behavior micro-modes by using unsupervised clustering, and constructing a behavior portrait library. Through multi-modal time sequence modeling and a self-adaptive space-time attention mechanism, behavior characteristics, an interface area and a report text are deeply fused, a multi-level correlation probability is output, and high-precision automatic tagging of content and an image area is realized.

The invention provides a visual fatigue relieving method based on ambient light self-adaption and an AI algorithm, and relates to the technical field of visual health protection. The visual fatigue relieving method based on ambient light self-adaption and the AI algorithm comprises the following specific steps: S1, data acquisition: acquiring a user eye image sequence in real time through a high-frame-rate camera; and S2, parameter extraction: when the user uses the eye for the first time, testing the eyes by combining the adjustable light source with ambient light. Multi-dimensional features (such as eyeball movement, blinking mode, pupil function and the like) of eyes are collected in real time through a high-frame-rate camera, an individualized base line is established and a fatigue rate value is dynamically calculated in combination with data of an ambient light sensor, and conversion from passive response to active prevention is realized. Through a hierarchical intervention strategy (such as brightness adjustment, blue light control and forced rest), the visual load is remarkably reduced, the intervention efficiency is improved, and the problem of insufficient adjustment hysteresis and individual adaptability in the prior art is solved.

Examples relate to computer-implemented methods for analyzing communication in digital evaluation. A computing device accesses multimodal data comprising video and audio information of human subjects and configures a computational model using this data to identify patterns in communication that correlate with assessment metrics. The configuring implements processing techniques that preserve relationships between features across different modalities. When a video recording of a candidate is received, the computing device processes the video using the configured computational model to extract communication features. These features may include facial expressions, gestures, eye movements, posture, vocal tone, and speech patterns. The device generates an evaluation of the candidate based on the extracted communication features and outputs a representation of the evaluation.

The invention belongs to the field of medical artificial intelligence, and particularly relates to a multi-modal data acquisition and fusion method for Alzheimer's disease. The method comprises the following steps: firstly, synchronously acquiring eye movement, expression, voice, gait and grip strength data of a subject through a virtual reality multi-task normal form, and combining with an MoCA scale to score a result; then preprocessing and feature extraction are carried out on each modal data, and unified feature representation is constructed; on the basis, a cross-modal attention mechanism is adopted to realize interaction and weighted fusion of multi-modal features, and a unified fusion feature vector table is generated; and finally, outputting structured data organized according to task fragments for auxiliary evaluation and modeling of cognitive impairment. The method can effectively solve the problems that in the prior art, single-mode information is insufficient, and multi-mode data are difficult to align and fuse, has the advantages of being low in cost, easy to popularize and high in detection accuracy, and can be widely applied to early recognition and auxiliary diagnosis of the Alzheimer's disease.

The invention relates to a hearing aid intelligent noise reduction and human voice enhancement system based on electroencephalogram signals, and belongs to the field of biomedical engineering and acoustic signal processing. The system comprises an electroencephalogram signal acquisition module, a multi-channel acoustic sensor array, an embedded neural signal processor, an adaptive beam forming module, a dynamic speech enhancement engine and a dual-mode output device, and constructs electroencephalogram-acoustics joint features by extracting an alpha / theta wave power ratio, a P300 component and auditory cortical Gamma phase synchronism. A deep network is driven to separate target voice, a wave beam direction and a frequency response curve are dynamically adjusted based on neural feedback, a closed-loop calibration unit is innovatively adopted, gain is reversely adjusted according to N1-P2 wave amplitude, heart rate variability and eye movement data are fused to optimize decisions, and when the signal-to-noise ratio is-5dB, the voice recognition rate reaches 89%, the auditory fatigue is reduced by 37%, and the decision conflict rate is smaller than 6%. The defects of attention blind area, noise separation failure and physiological adaptation of a traditional hearing aid are overcome. The system is suitable for the fields of hearing impairment rehabilitation, special communication and intelligent cabins.

The invention belongs to the technical field of aviation safety management and pilot efficiency evaluation, relates to a pilot multi-task processing efficiency evaluation method under a complex situation, and aims to solve the problem that a traditional method does not comprehensively consider a real flight environment and has a blank in the field of multi-task processing efficiency evaluation. The method comprises the steps that flight parameter data, eye movement data and task data of a pilot during flight are acquired and preprocessed; screening and calculating from three dimensions of attention distribution, work memory and task switching and burst task processing efficiency to obtain a key parameter data set; performing flight stage division based on flight parameter data in combination with the flight stage transfer atlas and the weighted multi-dimensional matching distance; presetting a weight matrix according to cognitive requirements of different flight stages; and matching a preset weight matrix according to a flight stage division result, and carrying out weighted fusion on each key parameter value to obtain a final performance evaluation value. In combination with multi-source data analysis, efficient evaluation of pilot multi-task processing efficiency is realized.

The invention, which relates to the technical field of artificial intelligence, discloses an artificial intelligence-based education evaluation and feedback system comprising a data acquisition module, a vector generation module, a prediction module, an error region positioning module and a feedback module. The system constructs a unified high-dimensional cognitive state vector by collecting answering behaviors, eye movement tracks, facial micro-expressions, voices and intonations and electroencephalogram signals of students; generating a learning evolution path map based on a dynamic Bayesian network and a causal reasoning mechanism, and predicting future learning bottleneck nodes; an error region is recognized through semantic deconstruction and graph matching, and context-associated personalized feedback content is generated in combination with a generative language model; according to the system, an evaluation feedback closed loop of cognitive state modeling, accurate identification of an erroneous region and intelligent feedback pushing is realized, and the accuracy of education evaluation and the effectiveness of intervention are improved.

The invention provides a community open space psychological recovery effect assessment method and system based on multi-modal perception. The method comprises the following steps: acquiring a physiological signal, a movement track, eyeball movement data and a panoramic image of a user; analyzing the image, extracting environmental element space distribution parameters, and calculating an environmental information entropy value; according to the physiological signal, calculating physiological stress deviation as a first type of error, analyzing the matching degree of a moving track and an environment structure as a second type of error, and combining eye movement characteristics and visual attraction distribution to calculate attention deviation as a third type of error; inputting the three types of errors and entropy values into an evaluation model, and outputting a recovery efficiency index and a multi-dimensional index representing error mitigation; and associating the environment parameters with the multi-dimensional indexes, explaining and extracting an environment intervention critical value and a recovery effect function, and generating a space optimization evaluation report in combination with dynamic indexes. According to the method, quantitative evaluation and accurate optimization decision support of the psychological recovery effect of the open space of the community are realized.

The invention belongs to the technical field of analog simulation, and particularly discloses an intelligent agent-based dynamic confrontation simulation system, which comprises a data acquisition module, an intelligent analysis module, a decision generation module, an intelligent agent behavior self-adaption module, a training evaluation module and a multi-mode man-machine interaction module, physiological, action, voice and eye movement data of trainees are collected in real time through a multi-modal sensor, and a tactical intention is recognized and a dynamic three-dimensional battlefield situation thermodynamic diagram is generated in combination with virtual battlefield environment parameters; an agent coping strategy is generated based on reinforcement learning and a decision tree, and an agent is driven to carry out real-time confrontation; and performing multi-dimensional quantitative evaluation on the whole training process through a training evaluation module, and performing closed-loop optimization on an agent decision and strategy library based on an evaluation result. The method supports various natural interaction modes such as voice, gestures and eye movement, remarkably improves the fidelity, intelligence and training efficiency of simulation training, and is suitable for the field of military training and tactical drilling.

The invention relates to the technical field of computers, and discloses an interaction control method of intelligent glasses. The method comprises the following steps: synchronously acquiring multi-modal data such as eye movement, voice, gestures and head postures and environment and application context information; carrying out independent time sequence feature coding on each modal data; generating a modulation vector in combination with the context, and outputting probability distribution of user intentions through a cross-modal attention fusion network; and a unique execution instruction is determined through an instruction arbitration module based on rules and a state machine. The system comprises corresponding function modules. According to the method, the accuracy, robustness and naturalness of interaction are improved through multi-modal synchronous fusion and a context self-adaption mechanism.

The invention provides an intelligent interaction system and method based on a multi-stage cognitive mode, and the system comprises a multi-modal data collection module which is used for collecting user interaction data through a multi-modal sensor, and the data comprise language input, non-language behaviors, interface operation data, expressions, eye movement tracks and the like; and the cognitive feature analysis module is used for calling a deep learning model to perform feature extraction on the interaction data. According to the method, language, behavior, interaction, physiology and other data are fused through the multi-modal sensor, the cognitive driving vector is generated by using the deep learning model, the real-time cognitive state of the user is effectively captured, then the probability distribution of the cognitive stage is constructed in combination with Bayesian reasoning, the problems that in the prior art, only the cognitive level can be statically judged, and real-time updating is difficult are solved, and the user experience is improved. The accuracy and timeliness of user state perception are remarkably improved, and dynamic accurate recognition and continuous modeling in the cognitive stage are achieved.

The invention discloses a driver fatigue state real-time identification system and method based on multi-modal deep learning, and relates to the technical field of fatigue driving detection. Firstly, feature extraction is performed on brain wave shapes and eye movement coordinates, and respective weights are calculated by using an attention mechanism, so that dynamic distribution of different modal features is realized. And then, in-vehicle illumination data is introduced to establish a credibility mapping function so as to carry out adaptive correction on an eye movement weight, thereby effectively reducing interference of a complex illumination environment on an identification result. And carrying out weighted splicing on the corrected multi-modal features, mapping the multi-modal features into a brain-eye collaborative fatigue value, and carrying out judgment in combination with the duration, so as to finally realize stable and reliable early warning control. The method has the advantages of high fusion precision, high environmental adaptability and low false alarm rate while ensuring the real-time performance, and the driving safety guarantee capability can be remarkably improved.

The invention belongs to the technical field of nerve regulation and control engineering, and provides a cerebrospinal fluid rhythm-based Alzheimer's disease electrical stimulation system, which is characterized in that first frequency band range data and second frequency band range data in brain wave signals are coupled to obtain cerebrospinal fluid rhythm data; then, calculating the phase difference between the first frequency band range data and the cerebrospinal fluid rhythm data; finally, the stimulation frequency is dynamically controlled according to the phase difference, and the stimulation current is dynamically controlled according to the brain tissue water volume fraction. The problem that specific metabolism requirements of non-fast eye movement sleep and fast eye movement sleep cycles cannot be matched by adopting single-frequency-band stimulation is solved, dynamic changes of post-traumatic encephaledema and requirements and influences of cerebrospinal fluid rhythm on current control are considered, the real-time adaptive capacity to the dynamic changes of the post-traumatic encephaledema is improved, and the application prospect is wide. And the treatment effect of the Alzheimer's disease is ensured.

The invention discloses a hidden multi-mode brain state real-time monitoring and closed-loop intervention system, and relates to the technical field of intelligent wearable equipment. The system comprises a cap body, and further comprises a multi-point electrode array used for collecting EEG (electroencephalogram) signals, EOG (eye movement) signals and EMG (electromyography) signals of a human body; the brain blood oxygen monitoring module is used for monitoring brain blood oxygen saturation; the signal receiving and processing module is used for preprocessing and analyzing data acquired by the multi-point electrode array and the brain blood oxygen monitoring module; the multi-mode stimulation module is used for implementing sensory and nerve stimulation on the user according to a judgment result of the state judgment module so as to intervene in a drowsiness or attention decline state; and the power supply module and the communication unit are used for providing a working power supply for the modules needing power supply in the system and carrying out data communication with external terminal equipment. The system can monitor the waking degree of the user with high precision and carry out closed-loop feedback intervention by various stimulation means.

The invention relates to a device for risk assessment of nervous system diseases, and a processor of the device is configured to receive eye movement data and electroencephalogram data which are associated in a time sequence, and fuse a double-branch deep neural network in a model through multiple modes to assess the risk of the nervous system diseases, performing feature extraction and attention mechanism-based cross-modal feature fusion on the eye movement data and the electroencephalogram data to obtain cross fusion features; and outputting risk assessment results and risk assessment parameters of at least one type of nervous system diseases through an output layer of the multi-modal fusion assessment model. According to the method and the device, the problem that diagnosis sensitivity and universality are insufficient due to the fact that diagnosis only depends on a single information source in related technologies is solved, disease features can be more comprehensively constructed from two levels of neural activity and behavior response by fusing data of two modes of eye movement and electroencephalogram, parallel diagnosis is further performed for multiple types of diseases, and the diagnosis efficiency is improved. The sensitivity and applicability of neuropsychiatric diagnosis are improved, and particularly, the method has more remarkable accuracy in early stages such as mild cognitive impairment and the like.

A drowsy driving detection method comprises: acquiring a side face image of a currently seated driver collected by a camera module; performing face recognition on the side face image to obtain side face feature parameters, and determining, according to the side face feature parameters, whether an ID file corresponding to the currently seated driver exists in a driver ID library; and if yes, periodically acquiring a side face image of the driver in the current period collected by the camera module, obtaining eye movement feature parameters of the driver in the current period according to the side face image of the current period, and determining whether the driver is driving while drowsy according to a comparison result between the eye movement feature parameters of the current period and the normal eye movement feature parameters of the driver.

The invention discloses an electroencephalogram data processing method for electroencephalogram glasses, which comprises the following steps: acquiring original electroencephalogram signals acquired by the electroencephalogram glasses, synchronously acquiring multi-mode auxiliary signals such as eye movement, acceleration and myoelectricity, and monitoring and acquiring quality parameters in real time; forming an initial data set containing the original electroencephalogram signal, the multi-mode auxiliary signal and a quality mark; taking the initial data set as input, and outputting a processed data set containing pure electroencephalogram signals and neural activity features through preprocessing, multi-modal artifact separation and personalized feature extraction in sequence; and packaging the processed data set and processing process meta-information thereof into a standard format file, executing local structured storage and cloud synchronous storage, recording a data operation log through a block chain technology, ensuring data security through an encryption technology, and completing whole-process data management.

The invention provides a continuous neural state monitoring method based on brain-computer interaction, and relates to the technical field of brain-computer interaction. The method sequentially comprises the steps that electroencephalogram, electro-oculogram and body movement signals are obtained and preprocessed, and a multi-channel time sequence segment and artifact marks are generated; constructing a target state scale sequence in combination with task geometry, physiological prior and eye movement events; performing time delay estimation and forward alignment on the input and the scale; setting space-time consistency constraint and drift penalty in the multi-scale state space model, and training to obtain a neural state mapping parameter set; extracting a session invariant subspace based on historical session data and executing small-step increment updating to form a multi-scale state space model subjected to individualized updating; and outputting a neural state vector and a confidence interval in model inference, generating a continuous monitoring result with a time index and an abnormal prompt, and storing the continuous monitoring result. According to the invention, high-real-time, high-stability and high-reliability continuous monitoring of non-invasive brain-computer interaction is realized.

The invention discloses an AI-based creative thinking auxiliary generation method and system, relates to the technical field of artificial intelligence, and solves the problem of inaccurate sound and picture matching in a traditional method by performing timestamp alignment and feature extraction on audio data and a visual image frame and calculating the correlation between the audio data and the visual image frame by using a cross-modal attention mechanism. According to the method, user eye movement track data is introduced, real attention points of a user are mapped into a visual sequence, and an optimized weight matrix is generated by constructing attention masks and fusing model attention weights, so that a generation result is more in line with perception key points of the user. And meanwhile, a feedback mechanism is established based on the synchronization error score, and when the sound and the picture are detected to be asynchronous, the visual frame timestamp can be dynamically adjusted, so that the self-adaptive correction of the content is realized. On the whole, the method has remarkable advantages in the aspects of improving modal alignment precision, enhancing user perception consistency and optimizing generation result naturalness.

The invention belongs to the field of virtual reality teaching application, and provides a teaching atmosphere automatic adaptation method and system in an immersive teaching environment. The method comprises the following steps: acquiring an eye movement image; recognizing a fixation point; carrying out ROI tracking; carrying out ROI boundary fusion; adaptively optimizing the object; adjusting the brightness of the ROI; and watching object interaction. According to the method, the immersion and interactivity of a future classroom can be improved, the use experience of an immersive virtual environment is facilitated, and deep fusion of an intelligent teaching environment and self cognition of a user is promoted.

The invention discloses a Parkinson's disease assessment method and system based on face video multi-modal physiological feature fusion, and relates to multiple technical fields of computer vision, physiological signal processing and the like, and the method comprises the following steps: S1, based on a continuous face video stream, extracting rPPG signals; then heart rate variability key parameters are calculated, and heart rate variability characteristics are obtained; s2, analyzing the dynamic change of an eye fixation point in the face video based on IPAST, extracting key eye movement behavior parameters, and obtaining eye movement behavior characteristics through a convolution gating loop unit; and S3, inputting the heart rate variability characteristics and the eye movement behavior characteristics into a multi-modal fusion network structure, and outputting a continuous risk score or an illness state label for assisting a doctor in early Parkinson risk assessment. According to the method, multiple physiological signals acquired through videos are deeply integrated, a multi-modal feature collaborative analysis framework is constructed, the subjective limitation of traditional scale evaluation is broken through, and the one-sidedness defect of single biomarker detection is overcome.

The invention provides a multi-mode interaction method, system and device based on an intelligent cabin and a vehicle, and relates to the technical field of intelligent cabins, and the method comprises the steps: displaying a main interface comprising a multi-mode interaction instruction setting control, a user authority management control and a real-time feedback display area; and flexible configuration and real-time feedback of the interaction mode of the intelligent cabin are realized. According to the method, the safety and personalized setting of operation authorities of different users are ensured by utilizing cross validation binding of the biological characteristic data and the authority levels. Meanwhile, through fusion processing of multi-mode input signals such as voice, gestures, eye movement and facial expressions, the weight of each signal is dynamically adjusted according to a signal priority rule, an interaction instruction conforming to the intention of the user is generated, and the naturalness and efficiency of interaction are improved. Finally, the execution mechanism is controlled to complete the corresponding operation by performing matching verification on the interaction intention instruction and the user permission level, and the safety and accuracy of the operation are further ensured.

The invention relates to the cross technical field of biomedical engineering and psychotherapy, in particular to an automatic control system for classified exposure therapy of phobia based on naked eye 3D. The system comprises an electroencephalogram signal feature extraction module, a heart rate variability feature extraction module, a skin electric response feature extraction module, an eye movement feature extraction module, a physiological load state feature extraction module, an exposure therapy process feature extraction module, a first feature fusion module, a second feature fusion module and a stimulation parameter adaptive control module. The method combines multi-mode biological signal real-time analysis and naked eye 3D stimulation parameter self-adaptive adjustment, and is suitable for clinical psychological treatment mechanisms and psychological health intervention scenes.

The invention provides a concentration training method and device based on visual tracking, and the method comprises the steps: collecting the real-time eye movement data of a user, including the staring point coordinate and the eye movement speed, and initializing the target trajectory of a training task; calculating a concentration deviation between the current gazing point and the target trajectory, and dynamically adjusting the target trajectory through nonlinear mapping based on the deviation to generate a dynamic trajectory; constructing a multi-feature fusion distraction scoring model based on the concentration deviation and the change trend of the concentration deviation in combination with the geometric features of the trajectory, and detecting and marking distraction events; when a distraction event is detected, self-adaptive visual correction stimulation is executed at the event position immediately, and a user is guided to return to a task; and after the training period is finished, dynamically adjusting the task difficulty coefficient of the next period according to the frequency and the average deviation of the distraction events in the period.