204 results about "Fixation point" patented technology

According to an aspect of the present invention, there is provided an electrical audio signal processing system and device, comprising: a computer graphics processing and selective visual display system with a screen; an eye tracking device; a processor; one or more computer memory devices; wherein the processor is arranged for operations comprising: measuring the user's eye movements to ascertain the specific word on which the user is fixated, by the eye tracking device; modifying the display at the user's current fixation point; applying a delay between the presentation of successive graphic elements based on the user's calculated rate to accommodate the user's required time; and presenting elements to the user at a rate based upon the user's required time.

The invention relates to the technical field of data processing, and particularly provides an operation intention recognition method, system and device based on multi-mode fusion and a medium, and the method comprises the steps: synchronously collecting interaction data of at least two modes of a user, the modes comprising at least two of gestures, voice and eye gaze; carrying out alignment processing on the interaction data, wherein the alignment processing comprises time synchronization and space mapping to a unified coordinate system; recognizing structured semantic information from each piece of aligned modal data, wherein the structured semantic information comprises a gesture type, a voice text and a fixation point coordinate; and based on a preset semantic rule and context memory, performing semantic association and anaphora resolution on the structured semantic information to obtain an operation intention. The method effectively overcomes the inherent defects of unnatural single-mode interaction, easy ambiguity and poor fault tolerance.

The invention relates to the field of electronic photo frames, and discloses an electronic photo frame intelligent interaction control method and system based on eye movement tracking, and the method comprises the steps: carrying out the real-time tracking of an eyeball position based on the eye image data of a user, and obtaining an eye movement track sequence according to the pupil center coordinate change and the eyelid opening and closing state change; calculating a fixation point coordinate, a fixation time length and an eye hopping frequency based on the eye movement track sequence; the current eye movement behavior type of the user is obtained through a personalized recognition model; matching with a preset eye movement pattern library based on the current eye movement behavior type of the user to obtain a candidate instruction set; and executing photo frame function operation according to the control instruction. The method has the following effect that the convenience and accuracy of man-machine interaction are improved based on non-contact intelligent control of eye movement.

The invention belongs to the field of eye movement tracking, and particularly relates to an eye movement tracking system, method and equipment integrating electroencephalogram signals and a non-contact eye tracker. The objective of the invention is to solve the problems of limited acquisition range of a non-contact eye tracker and low spatial resolution of eye movement behavior deduction by EEG signals in the prior art. The method comprises the steps that eye movement data and electroencephalogram signals are synchronously collected, after calibration mapping and time sequence alignment preprocessing are conducted, an electroencephalogram modal behavior recognition model is supervised and trained through eye movement fixation point labels, and the incidence relation of the two modals is established. A dynamic fusion strategy is adopted in the online stage, when eye movement data are valid, a model prediction result and the eye movement data are fused to output a fixation point, when eye movement fails, the fixation point is calculated in combination with model prediction and historical data, enhanced compensation of eye movement tracking by electroencephalogram signals is achieved, the defect that the spatial resolution of a single EEG signal is low is effectively overcome, and the accuracy of eye movement tracking is improved. And the robustness and the tracking precision of the system in a complex scene are improved.

The invention discloses an intelligent glasses image adjusting system based on eye movement tracking and gesture fusion, and relates to the technical field of intelligent equipment. A multi-modal sensing module is arranged to construct a multi-modal sensing layer to capture eyeball movement tracks and gesture actions; a fixation point prediction module is set to process a dynamic scene through a space-time attention mechanism to obtain a fixation point prediction area, a gesture semantic understanding module is set to process gesture actions based on a Transform architecture, and the gesture actions of a user are converted into image adjustment instructions. An image enhancement strategy setting module designs a multi-stage image enhancement strategy according to the fixation point prediction area and the image adjustment instruction, and sets a dynamic adjustment intensity control module to perform adaptive adjustment to obtain a dynamic adjustment intensity control result; an eye movement-gesture cooperative control module is arranged to provide an eye movement-gesture cooperative control mechanism to realize image area selection and parameter adjustment, and accurate image area selection and parameter adjustment are realized.

The invention discloses an AR glasses control system and method, and belongs to the field of artificial intelligence and AR glasses application, and the method comprises the steps: capturing a user face image, combining a voice assistant, collecting user voiceprint data, and carrying out the matching through a recurrent neural network; constructing a three-dimensional environment model, positioning and marking obstacles and interactive objects, and identifying objects in the environment; the method comprises the following steps of: tracking a gesture, acquiring a key point coordinate by adopting an OpenPose algorithm, identifying a predefined gesture by utilizing a support vector machine, and mapping the gesture into a specific operation instruction; analyzing the user intention based on natural language processing, and calling a corresponding function module according to an analysis result; acquiring a gazing direction of a user by using an eye movement tracking sensor, calculating a screen focus position through an algorithm, and dynamically highlighting options or displaying information according to a gazing point position; through multi-modal data fusion, gestures, voices and fixation point input are analyzed in a unified manner. By accurately understanding the intention of the user, the interaction precision of the AR glasses is improved, and the user experience is optimized.

The invention discloses a virtual reality interaction method and system applied to classical famous picture display, and particularly relates to the technical field of virtual reality interaction. Multi-dimensional high-definition image acquisition and composition element three-dimensional modeling are performed on a painting, a cultural context label system is constructed in combination with historical data, a fixation point, an action posture and a staying behavior of a user are perceived in real time, accurate matching of a user behavior and a semantic label is realized, and multi-modal interaction feedback content consistent with the style of the painting is generated, so that the user experience is improved. And typical paths and misunderstanding areas are identified through clustering analysis of multi-user behavior data, and context tags and response logic are dynamically optimized, so that personalized recommendation and semantic guidance are realized, the understanding depth and immersion experience of the users on classical art are effectively improved, and the cultural expression ability and intelligent adaptability of the system are enhanced.

The invention discloses a first view angle video generation method and device based on a mask diffusion model and fixation point constraint, and the method comprises the steps: constructing an end-to-end deep learning frame for the demands of video filling, prediction and visual attention region control in the generation of a first view angle video; the diversified first-view-angle video generation conforming to the visual logic is realized. The method comprises the following steps: firstly, dividing an input video into a condition frame set and an unknown frame set, and adding random noise to an unknown frame by using a dynamic mask module I to generate a noisized frame; performing reverse de-noising generation on the video through a full 3D convolutional neural network module II, and guiding the network to generate video content conforming to a space-time law by taking a diffusion step and a fixation point track as conditional constraints; and finally, saliency prediction is carried out by using a fixation point positioning module III. By jointly optimizing the loss of the reverse denoising generation process and the loss of the fixation point probability graph, the reasonability and diversity of the generated video are further improved. According to the method, through joint optimization of the mask diffusion strategy and fixation point constraint, the space-time continuity, the anti-noise capability and the adaptability to the fixation point trajectory of the generated video can be effectively improved, and the method is particularly suitable for a complex multi-face expression interaction scene.

The invention provides an on-line examination anti-cheating method and system based on AI fusion sight tracking, and relates to the technical field of on-line education safety, and the method comprises the steps: extracting a head motion feature sequence of an examinee, calculating the coordinates and duration of a sight fixation point, analyzing the fixation behavior of a functional region, and recognizing a periodic jump mode in a fixation migration feature map. The gazing mode mutation point is identified by combining autoregression spectrum analysis and wavelet analysis, so that the cheating behavior is accurately identified, the safety and fairness of the online examination are effectively improved, and the misjudgment rate is reduced.

The invention discloses a Parkinson's disease assessment method and system based on face video multi-modal physiological feature fusion, and relates to multiple technical fields of computer vision, physiological signal processing and the like, and the method comprises the following steps: S1, based on a continuous face video stream, extracting rPPG signals; then heart rate variability key parameters are calculated, and heart rate variability characteristics are obtained; s2, analyzing the dynamic change of an eye fixation point in the face video based on IPAST, extracting key eye movement behavior parameters, and obtaining eye movement behavior characteristics through a convolution gating loop unit; and S3, inputting the heart rate variability characteristics and the eye movement behavior characteristics into a multi-modal fusion network structure, and outputting a continuous risk score or an illness state label for assisting a doctor in early Parkinson risk assessment. According to the method, multiple physiological signals acquired through videos are deeply integrated, a multi-modal feature collaborative analysis framework is constructed, the subjective limitation of traditional scale evaluation is broken through, and the one-sidedness defect of single biomarker detection is overcome.

The invention relates to the technical field of large model cheating detection, in particular to an online interview large model cheating detection system and method based on eye movement tracking, and the detection method comprises the steps: S1, system initialization and environment adaptation; s2, eye movement tracking calibration and baseline establishment; s3, collecting and preprocessing real-time eye movement data; s4, executing an anomaly detection algorithm; and S5, carrying out risk assessment and outputting a result. According to the scheme, based on the WebGazer.js technology, the mapping relation between the fixation point and the screen coordinates is established through nine-point initialization calibration, and the personalized coefficient k is generated in combination with the pupil diameter and the facial features of the candidate, so that the eye movement tracking error is greatly reduced, and the large model cheating detection precision requirement can be met without special hardware. The design not only reduces the deployment cost, but also breaks through the hardware limitation, so that the technology can adapt to mainstream notebook equipment, is convenient for large-scale popularization and application, and solves the contradiction that high precision and low cost cannot be achieved at the same time in the traditional scheme.

The invention discloses an attention monitoring method, device and equipment based on multi-modal sensing and a medium, and relates to the technical field of attention monitoring. The method comprises the following steps: collecting eye movement image data of eyeballs of a user; extracting fixation point stability features, eye movement track features, blink mode features and pupil dynamic features based on the eye movement image data; determining a concentration score of the user based on the fixation point stability feature, the blink mode feature, the pupil dynamic feature and the eye movement track feature; and if the concentration score is lower than a preset threshold value, sending reminding information to a user through the wearable device. The attention distraction behavior of the user in a continuous high attention demand scene can be accurately identified and prompted in time.

The invention discloses an English classroom attention regulation and control method combined with brain wave detection, and relates to the technical field of English classroom attention regulation and control, and the method comprises the steps: obtaining a multi-channel brain wave signal of a student in English classroom learning through a brain wave collection device, collecting visual gazing track data through an eye tracker, and obtaining a brain wave signal; capturing a face image sequence through a camera; performing frequency band separation on the multi-channel electroencephalogram signal, extracting energy of a theta wave and a beta wave of a prefrontal lobe, and calculating a theta / beta power ratio; performing de-noising and clustering processing on the visual gaze track data, identifying an effective gaze interval, calculating a reciprocal of a gaze point distribution variance in unit time, generating a visual focusing stability parameter, performing action unit identification on a facial image sequence, extracting smile frequency, eyebrow intensity and blink period characteristics, and outputting a cognitive emotion state score; and constructing a multi-modal attention fusion evaluation model based on the theta / beta power ratio, the visual focusing stability parameter and the cognitive emotion state score.

The invention relates to the technical field of intelligent visual detection, in particular to visual field detection equipment, system and method based on extended reality (XR), and the method comprises the steps that an image feature extraction module collects image data in real time, extracts features and eliminates optical distortion; the fixation deviation correction module analyzes eye movement data in real time, quantifies fixation deviation and triggers visual prompt and correction; the adaptive stimulation generation module dynamically encrypts to generate a test point image and adaptively adjusts the test point image according to visual field analysis and test requirements; the dynamic adjustment image module predicts and optimizes a test point presentation sequence according to image features and fixation points, automatically encrypts a high-probability defect area and quickly jumps to a key screening area; and the visual field defect identification module extracts image features, calculates the defect probability of each region in real time by using a Bayesian network in combination with subject response data, and locates a high-risk region. The system provides an efficient and accurate visual field detection scheme through the synergistic effect of multiple modules.

The invention discloses an image display method for myopia prevention and control and a related product thereof. The image display method for myopia prevention and control comprises the following steps: determining a sight fixation point when a user watches an electronic display screen; determining an out-of-focus fuzzy display area on the electronic display screen according to the sight fixation point; and performing blurring processing for simulating a myopia defocusing effect on an image positioned on the defocusing blurred display area so as to obtain an image with the myopia defocusing effect on the retina of the user. By utilizing the method disclosed by the invention, the myopia prevention and control effect can be obtained under the condition that the myopia patient does not wear the frame glasses with the myopia defocus effect, so that the myopia prevention and control cost of the user is reduced.

The invention relates to the technical field of image data processing, in particular to a method for improving the detail display quality of VR equipment by using a binocular difference mechanism. The method comprises the following steps: acquiring a binocular image of a current frame of VR equipment and a user watching behavior feature; the user gazing behavior characteristics comprise a gazing area and a gazing point; according to the disparity map of the binocular image, determining the disparity significance of the fixation area; determining an initial brightness interval and a binocular brightness difference degree according to a brightness order of a corresponding point of the fixation point in the binocular image; determining a brightness order adjustment weight according to the parallax saliency, the binocular brightness difference degree and the user watching behavior characteristics; adjusting the initial brightness interval according to the brightness order adjusting weight to obtain an adjusted brightness interval; and adjusting the brightness of the binocular image according to the adjusted brightness interval, and obtaining a binocular image currently output by the VR equipment based on the adjusted binocular image. The problem of display delay is avoided, and the quality of display details is ensured.

The invention provides a machine learning and deep learning combined calibration-free eye movement tracking system and method, and the system comprises the following modules: a pre-calibration module which is used for collecting a near-to-eye image of the eyes of a user through a near-to-eye camera and an infrared light source, extracting pupil center coordinates and cornea reflection point coordinates, and outputting the pupil center coordinates and the cornea reflection point coordinates; calculating a personalized kappa angle of the user based on the extracted pupil center coordinates and cornea reflection point coordinates; the decision maker module adopts an AdaBoost ensemble learning algorithm, classifies the users according to the kappa angle obtained through calculation, and outputs a corresponding weight combination; and the deep learning module is used for receiving the output weight combination and the real-time near-eye image, and predicting the coordinates of the fixation point of the user through a convolutional neural network and a residual connection structure. According to the invention, the application flexibility and user friendliness of the eye movement tracking technology are greatly enhanced while the precision is guaranteed.

The invention relates to the field of intelligent perception, and mainly relates to a user interaction intention perception method and system based on intelligent glasses, and the method comprises the steps: obtaining the abnormal degree of an eye jump drop point according to the main eye jump and the subsequent correction eye jump times after detecting that the current eye jump drop point of a user enters an interaction region of an interactive object; acquiring a pupil diameter change sequence after the main eye jump, and acquiring a pupil transient response score based on the pupil diameter change sequence; after a certain time from the falling point time of the main eye jump, acquiring a real-time double-variable contour ellipse area based on information of a plurality of latest fixation points, and acquiring a fixation stability score based on the real-time double-variable contour ellipse area; obtaining a real-time gaze residence time according to the falling point time of the main eye jump, and obtaining a residence time evaluation score based on the real-time gaze residence time; fusing the abnormal degree, the pupil instantaneous response score, the stability score and the evaluation score to obtain a dynamic interaction intention score of the user, and determining the interaction intention of the user based on the intention score. The perception accuracy of the user interaction intention is improved.

The invention discloses a focus depth determination method and a related device, and relates to the technical field of rendering, and the method comprises the steps: collecting eye movement data of a user according to a preset eye movement sampling rate in a process that the user watches a three-dimensional scene, so as to obtain multi-frame eye movement data collected in a target duration, and determining a focus depth of the user according to the multi-frame eye movement data, determining a fixation point position of the sight line of the user in the target three-dimensional model, generating a sphere with the radius as a target value by taking the fixation point position as a sphere center, obtaining a target vertex set in the sphere in the target three-dimensional model, and performing vertex density analysis on the target vertex set to obtain a focus depth for rendering the three-dimensional scene. According to the method, the focus depth dynamically adjusted along with the sight line of the user can be generated, the detail resolution capability of the user on the fixation point is improved, and the watching experience of the user on the three-dimensional scene is further improved.

The invention discloses a Rhatish culture VRAR virtual-real fusion interaction method based on eye movement tracking, and particularly relates to the technical field of virtual reality and augmented reality man-machine interaction. The method comprises the following steps: acquiring an eye image of a user through a near-infrared image acquisition unit, and inputting the eye image into a double-branch convolutional neural network model to estimate a two-dimensional fixation point of a screen; performing nonlinear correction on a fixation point based on the user personalized sight line offset correction model established in the offline calibration stage; in the online interaction stage, the corrected fixation point is combined with equipment pose data to be converted into three-dimensional space rays, the distance and angle relation between the three-dimensional space rays and an interaction target is calculated, and an interaction instruction is triggered when a threshold condition is continuously met. According to the method and the system, accurate, stable and natural eye movement interaction of the virtual contents of the Rhatish culture and the entity exhibits in the VR / AR environment is realized, and the user experience and the system robustness are improved.

The present invention belongs to the technical field of eye movement tracking, and discloses a near-eye fixation point tracking method, device and medium based on an event sensor, including: an infrared light source provides infrared illumination for the eye and the surrounding area of the eye, and the event sensor collects information on the eye and the surrounding area of the eye to obtain a series of event stream information generated by the change in light caused by movement; the time window clustering event set T is sent to the ROI and pupil calculation module to output pupil ellipse information; the event stream information is sent to the vector calculation module, and in combination with the pupil ellipse information, the movement vector of the pupil center is calculated; in combination with the movement vector, the pupil ellipse information is updated; in combination with the dynamic filtering method, the pupil ellipse information is smoothed; the filtered pupil ellipse information is sent to the line-of-sight regression and compensation model, and in combination with the deep learning method, the fixation point coordinates are calculated. The present invention introduces an event sensor and can more accurately identify moving targets in a dynamic scene.

The invention provides a fitting method of a 3D eye model and electronic equipment, and aims to improve the accuracy of the eye model. Comprising the following steps: in response to an eye model fitting instruction, obtaining eye images obtained when eyes of a user watch fixation points at different positions; obtaining eyeball parameters of the user by using the image position coordinates of the pupil center of the user in each eye image and the standard visual axis vector corresponding to each eye image; obtaining a current eyeball rotation angle based on the eyeball parameters of the user and the image position coordinates of the pupil center in the current eye image every specified duration; according to the current eyeball rotation angle and the eyeball parameters, current eyeball model parameters are obtained, and the eyeball model parameters comprise the position vector of the eyeball center, the position vector of the pupil center, the position vector of the curvature center and the visual axis vector; and updating a preset initial eye model by using the current eyeball model parameters of the user to obtain a current 3D eye model of the user.

The invention discloses an automobile cabin software control method and system based on multi-mode interaction, and relates to the field of multi-mode data, and the method comprises the steps: S1, multi-source signal synchronous collection and timestamp calibration; a user voice instruction, a gesture track and eye fixation point information are collected in real time through a distributed sensor array; according to sensor data, a three-level synchronization architecture is combined with a PTP precise time protocol to achieve microsecond-level clock synchronization, a unified time service timestamp is embedded in a data packet header, and the timestamp error is controlled within + / -0.5 ms; s2, dynamic environment interference compensation; the method comprises the following steps: constructing an environmental noise perception model; the construction of the environmental noise perception model comprises the following steps: starting a regional strong light suppression algorithm for a strong light scene with the illumination intensity greater than 10,000 lux; activating a motion compensation module for a bumpy scene with a vertical acceleration greater than 0.3 g, and outputting an environment compensation multi-modal data stream; s3, analyzing time-space association semantics; and S4, confidence decision and instruction execution.

The invention discloses an eye movement tracking stability enhancing method, system and device and a storage medium, and belongs to the field of human-computer interaction and computer vision. The method comprises the following steps: acquiring an eye image and an original sight fixation point coordinate; performing quantitative evaluation on the image from five dimensions of definition, contrast and the like to generate an eye image quality evaluation grade; calculating the moving speed and acceleration of the current point based on the historical sight fixation point sequence; fusing, and judging and correcting the original coordinates. Through multi-modal information fusion and hierarchical adaptive correction, noise jump caused by poor image quality can be accurately distinguished and filtered out, meanwhile, the real eye movement dynamic state is effectively reserved, and the output stability and reliability of an eye movement tracking system are remarkably improved.

The invention belongs to the technical field of pilot ability assessment, and particularly discloses a pilot ability assessment method based on dynamic time warping and hierarchical clustering, which comprises the following steps of: acquiring eye movement data of a tested pilot in a flight simulation task process, preprocessing the eye movement data, extracting behavior indexes of each stage of a flight task, and calculating a pilot ability assessment result; generating a fixation area number sequence according to the fixation point position; in the selection target evaluation stage, a dynamic time warping algorithm is used for carrying out nonlinear alignment on the gaze sequences of all the pilots, and an eye movement difference degree matrix between the pilots is generated; based on the difference degree matrix, adopting a hierarchical clustering algorithm to group the pilots; and outputting an ability evaluation result of the pilot according to the distribution of the pilot in the difference degree matrix and the deviation information of the pilot and the teacher watching sequence. According to the method, structured comparison and capability grade evaluation of complex cognitive behaviors can be realized, and an evaluation result has relatively high objectivity and interpretability.

The invention discloses a perioperative cognitive impairment predicting and monitoring device, relates to the technical field of medical instruments, and aims to solve the problem of low accuracy of perioperative cognitive impairment predicting and monitoring for children in the prior art. An electroencephalogram signal acquisition module, an eye movement tracking module, a behavior interaction module, a physiological parameter monitoring module and a data processing and analyzing module are integrated in the head-mounted VR glasses; the electroencephalogram signal acquisition module is used for acquiring an electroencephalogram signal of a child, processing the electroencephalogram signal and then transmitting the electroencephalogram signal to the data processing and analyzing module; the eye movement tracking module is used for capturing motion images of eyeballs of children, obtaining fixation points, eye jump frequencies and pupil diameter change parameters of the eyeballs and transmitting the parameters to the data processing and analyzing module. The method has the advantage that the accuracy of predicting and monitoring the cognitive impairment in the perioperative period of children is improved.

The invention relates to the field of music visual playing analysis, in particular to a music visual playing analysis system based on an eye movement tracking technology, which comprises an eye movement data acquisition module, a data preprocessing module, a visual feature extraction module, a music score synchronous mapping module, a performance evaluation module and a user feedback module. According to the scheme, a feature extraction method based on a video converter is adopted, eyeball movement data of a player is finely analyzed, and feature data of gaze point distribution, gaze time and a glancing path are extracted for subsequent deep analysis; the OCR technology is introduced to identify symbols in a music score image, and a dynamic mapping method is combined to accurately match eyeball movement data of a player with the symbols in the music score image, so that the method not only can monitor eye movement behaviors of the player in real time, but also can associate the behaviors with specific music score contents, thereby improving the accuracy of the music score. And teachers are helped to understand the visual playing process of students more deeply.

The invention provides an eye movement trajectory analysis method and system based on hybrid clustering and time constraint, and belongs to the technical field of computer vision. Calculating a time difference and a moving speed between continuous original eye movement data points; comparing the moving speed with a speed threshold value, and classifying the moving speed into candidate fixation points and glancing points; extracting spatial features and time features of the candidate fixation points, performing standardization processing, and performing weighted fusion to obtain spatial-temporal feature vectors; clustering the spatio-temporal feature vectors to obtain preliminary clustering labels of the candidate fixation points; performing time constraint processing on each cluster; calculating the duration of the clustering cluster after the time constraint processing, and generating a final clustering label of the candidate fixation point; and outputting a final classification label of each original eye movement data point in combination with the classification result of the glancing points.

The invention discloses a multi-task fixation point estimation method, which relates to the technical field of computer vision and deep learning, and comprises the following steps of: firstly, constructing a data set for providing associated data from tasks to labels and supporting multi-task cooperative training, and then constructing an MTHGaze model, through a semi-structured framework, deep learning, a geometric constraint fitting model and key tasks of cooperative processing are fused, a loss function is designed and trained, defects of each task are solved through targeted loss function design, and a scientific training process enables the model to be iteratively optimized. Finally, the targets of improving the estimation precision of the fixation point, ensuring the physical rationality and enhancing the environmental robustness are achieved; and finally, performing fixation point mapping. According to the estimation method, through data set construction, data consistency can be guaranteed, training noise can be reduced, a real reference can be provided for loss function calculation, error back propagation can be realized, and Euclidean distance errors can be minimized through L2 norms of real fixation point coordinates and prediction coordinates.

According to an aspect of the present invention, there is provided a computer graphics processing and selective visual display system, comprising: a computer graphics processing and selective visual display system with a screen; an eye tracking device; a processor; one or more computer memory devices; wherein the processor is arranged for operations comprising: measuring the user's eye movements to ascertain the specific word on which the user is fixated, by the eye tracking device; modifying the display at the user's current fixation point; applying a delay between the presentation of successive graphic elements based on the user's calculated rate to accommodate the user's required time; and presenting elements to the user at a rate based upon the user's required time.