113 results about "Fixation point" patented technology

The invention discloses an intelligent glasses image adjusting system based on eye movement tracking and gesture fusion, and relates to the technical field of intelligent equipment. A multi-modal sensing module is arranged to construct a multi-modal sensing layer to capture eyeball movement tracks and gesture actions; a fixation point prediction module is set to process a dynamic scene through a space-time attention mechanism to obtain a fixation point prediction area, a gesture semantic understanding module is set to process gesture actions based on a Transform architecture, and the gesture actions of a user are converted into image adjustment instructions. An image enhancement strategy setting module designs a multi-stage image enhancement strategy according to the fixation point prediction area and the image adjustment instruction, and sets a dynamic adjustment intensity control module to perform adaptive adjustment to obtain a dynamic adjustment intensity control result; an eye movement-gesture cooperative control module is arranged to provide an eye movement-gesture cooperative control mechanism to realize image area selection and parameter adjustment, and accurate image area selection and parameter adjustment are realized.

The invention discloses an attention monitoring method, device and equipment based on multi-modal sensing and a medium, and relates to the technical field of attention monitoring. The method comprises the following steps: collecting eye movement image data of eyeballs of a user; extracting fixation point stability features, eye movement track features, blink mode features and pupil dynamic features based on the eye movement image data; determining a concentration score of the user based on the fixation point stability feature, the blink mode feature, the pupil dynamic feature and the eye movement track feature; and if the concentration score is lower than a preset threshold value, sending reminding information to a user through the wearable device. The attention distraction behavior of the user in a continuous high attention demand scene can be accurately identified and prompted in time.

The invention relates to the field of intelligent perception, and mainly relates to a user interaction intention perception method and system based on intelligent glasses, and the method comprises the steps: obtaining the abnormal degree of an eye jump drop point according to the main eye jump and the subsequent correction eye jump times after detecting that the current eye jump drop point of a user enters an interaction region of an interactive object; acquiring a pupil diameter change sequence after the main eye jump, and acquiring a pupil transient response score based on the pupil diameter change sequence; after a certain time from the falling point time of the main eye jump, acquiring a real-time double-variable contour ellipse area based on information of a plurality of latest fixation points, and acquiring a fixation stability score based on the real-time double-variable contour ellipse area; obtaining a real-time gaze residence time according to the falling point time of the main eye jump, and obtaining a residence time evaluation score based on the real-time gaze residence time; fusing the abnormal degree, the pupil instantaneous response score, the stability score and the evaluation score to obtain a dynamic interaction intention score of the user, and determining the interaction intention of the user based on the intention score. The perception accuracy of the user interaction intention is improved.

The invention discloses a Rhatish culture VRAR virtual-real fusion interaction method based on eye movement tracking, and particularly relates to the technical field of virtual reality and augmented reality man-machine interaction. The method comprises the following steps: acquiring an eye image of a user through a near-infrared image acquisition unit, and inputting the eye image into a double-branch convolutional neural network model to estimate a two-dimensional fixation point of a screen; performing nonlinear correction on a fixation point based on the user personalized sight line offset correction model established in the offline calibration stage; in the online interaction stage, the corrected fixation point is combined with equipment pose data to be converted into three-dimensional space rays, the distance and angle relation between the three-dimensional space rays and an interaction target is calculated, and an interaction instruction is triggered when a threshold condition is continuously met. According to the method and the system, accurate, stable and natural eye movement interaction of the virtual contents of the Rhatish culture and the entity exhibits in the VR / AR environment is realized, and the user experience and the system robustness are improved.

The invention provides a fitting method of a 3D eye model and electronic equipment, and aims to improve the accuracy of the eye model. Comprising the following steps: in response to an eye model fitting instruction, obtaining eye images obtained when eyes of a user watch fixation points at different positions; obtaining eyeball parameters of the user by using the image position coordinates of the pupil center of the user in each eye image and the standard visual axis vector corresponding to each eye image; obtaining a current eyeball rotation angle based on the eyeball parameters of the user and the image position coordinates of the pupil center in the current eye image every specified duration; according to the current eyeball rotation angle and the eyeball parameters, current eyeball model parameters are obtained, and the eyeball model parameters comprise the position vector of the eyeball center, the position vector of the pupil center, the position vector of the curvature center and the visual axis vector; and updating a preset initial eye model by using the current eyeball model parameters of the user to obtain a current 3D eye model of the user.

The invention discloses an automobile cabin software control method and system based on multi-mode interaction, and relates to the field of multi-mode data, and the method comprises the steps: S1, multi-source signal synchronous collection and timestamp calibration; a user voice instruction, a gesture track and eye fixation point information are collected in real time through a distributed sensor array; according to sensor data, a three-level synchronization architecture is combined with a PTP precise time protocol to achieve microsecond-level clock synchronization, a unified time service timestamp is embedded in a data packet header, and the timestamp error is controlled within + / -0.5 ms; s2, dynamic environment interference compensation; the method comprises the following steps: constructing an environmental noise perception model; the construction of the environmental noise perception model comprises the following steps: starting a regional strong light suppression algorithm for a strong light scene with the illumination intensity greater than 10,000 lux; activating a motion compensation module for a bumpy scene with a vertical acceleration greater than 0.3 g, and outputting an environment compensation multi-modal data stream; s3, analyzing time-space association semantics; and S4, confidence decision and instruction execution.

The invention discloses an eye movement tracking stability enhancing method, system and device and a storage medium, and belongs to the field of human-computer interaction and computer vision. The method comprises the following steps: acquiring an eye image and an original sight fixation point coordinate; performing quantitative evaluation on the image from five dimensions of definition, contrast and the like to generate an eye image quality evaluation grade; calculating the moving speed and acceleration of the current point based on the historical sight fixation point sequence; fusing, and judging and correcting the original coordinates. Through multi-modal information fusion and hierarchical adaptive correction, noise jump caused by poor image quality can be accurately distinguished and filtered out, meanwhile, the real eye movement dynamic state is effectively reserved, and the output stability and reliability of an eye movement tracking system are remarkably improved.

The invention belongs to the technical field of pilot ability assessment, and particularly discloses a pilot ability assessment method based on dynamic time warping and hierarchical clustering, which comprises the following steps of: acquiring eye movement data of a tested pilot in a flight simulation task process, preprocessing the eye movement data, extracting behavior indexes of each stage of a flight task, and calculating a pilot ability assessment result; generating a fixation area number sequence according to the fixation point position; in the selection target evaluation stage, a dynamic time warping algorithm is used for carrying out nonlinear alignment on the gaze sequences of all the pilots, and an eye movement difference degree matrix between the pilots is generated; based on the difference degree matrix, adopting a hierarchical clustering algorithm to group the pilots; and outputting an ability evaluation result of the pilot according to the distribution of the pilot in the difference degree matrix and the deviation information of the pilot and the teacher watching sequence. According to the method, structured comparison and capability grade evaluation of complex cognitive behaviors can be realized, and an evaluation result has relatively high objectivity and interpretability.

The invention provides an eye movement trajectory analysis method and system based on hybrid clustering and time constraint, and belongs to the technical field of computer vision. Calculating a time difference and a moving speed between continuous original eye movement data points; comparing the moving speed with a speed threshold value, and classifying the moving speed into candidate fixation points and glancing points; extracting spatial features and time features of the candidate fixation points, performing standardization processing, and performing weighted fusion to obtain spatial-temporal feature vectors; clustering the spatio-temporal feature vectors to obtain preliminary clustering labels of the candidate fixation points; performing time constraint processing on each cluster; calculating the duration of the clustering cluster after the time constraint processing, and generating a final clustering label of the candidate fixation point; and outputting a final classification label of each original eye movement data point in combination with the classification result of the glancing points.

The invention discloses a multi-task fixation point estimation method, which relates to the technical field of computer vision and deep learning, and comprises the following steps of: firstly, constructing a data set for providing associated data from tasks to labels and supporting multi-task cooperative training, and then constructing an MTHGaze model, through a semi-structured framework, deep learning, a geometric constraint fitting model and key tasks of cooperative processing are fused, a loss function is designed and trained, defects of each task are solved through targeted loss function design, and a scientific training process enables the model to be iteratively optimized. Finally, the targets of improving the estimation precision of the fixation point, ensuring the physical rationality and enhancing the environmental robustness are achieved; and finally, performing fixation point mapping. According to the estimation method, through data set construction, data consistency can be guaranteed, training noise can be reduced, a real reference can be provided for loss function calculation, error back propagation can be realized, and Euclidean distance errors can be minimized through L2 norms of real fixation point coordinates and prediction coordinates.

The invention discloses an ADHD screening method and system based on mobile terminal eye movement tracking and a multi-modal agent. The method comprises the following steps: (1) collecting and preprocessing mobile terminal eye movement data; (2) performing fixation point estimation based on a lightweight MGazeNet model; (3) eye movement feature extraction based on a multi-normal form interaction task; and (4) carrying out comprehensive diagnosis analysis based on machine learning and a large model agent. The invention further provides an ADHD screening system based on mobile terminal eye movement tracking and the multi-modal intelligent agent. The ADHD screening system sequentially comprises the following modules: (1) a front-end data acquisition and preprocessing module; (2) an end side eye movement tracking reasoning module; (3) an interactive task control and feature calculation module; and (4) a multi-mode intelligent diagnosis and analysis module. The ADHD screening method effectively solves the problems that traditional ADHD screening depends on expensive professional equipment and subjective scales, so that the popularity rate is low, and objectivity is poor, screening portability and diagnosis objective accuracy are improved, and the intelligent level of ADHD auxiliary diagnosis and user interaction experience are improved.

The embodiment of the invention provides an emotion detection method based on a multi-mode electroencephalogram eye movement signal. According to the method, a dual-path electroencephalogram modal encoder is used for determining time sequence characteristics of an electroencephalogram in a time domain path, determining frequency domain characteristics of an amplitude spectrum of the electroencephalogram in a frequency domain path, and obtaining dual-path time domain and frequency domain fused electroencephalogram characteristics based on the time sequence characteristics and the frequency domain characteristics; the dual-path eye movement modal encoder is used for determining pupil features of the eye movement data in the pupil diameter path, determining eye movement fixation features of the eye movement data in the fixation point path, and obtaining fused eye movement features based on the pupil features and the eye movement fixation features; and the multi-modal expert mixing module is used for dynamically performing modal calling on the multi-modal mixed characteristics which are subjected to modal alignment and are fused with the electroencephalogram characteristics and the eye movement characteristics, performing emotion detection by utilizing an expert with a modal corresponding to the multi-modal mixed characteristics, and outputting an emotion classification result. According to the embodiment of the invention, alignment is realized through multi-task pre-training, and the depressive emotion is accurately recognized.

The invention discloses a sight tracking method and system based on cascade deep learning and distance correction, and a storage medium. The method comprises the following steps: processing an eye image through a cascade deep learning model, and obtaining pupil center coordinates and cornea reflection light spot coordinates of two eyes; calculating a binocular pixel distance as a distance correction factor based on the pupil center coordinates, and calculating a relative position difference vector based on the pupil and light spot coordinates; establishing a three-dimensional nonlinear polynomial regression model taking the relative position difference vector and the distance correction factor as input; obtaining calibration data through a multi-distance sampling strategy and solving a model coefficient; and in the real-time tracking stage, calculating the coordinates of the fixation point of the screen by using the solved model. According to the method, the distance correction factor and the three-dimensional mapping model are introduced, so that geometric errors caused by forward and backward movement of the head of the user are effectively eliminated, meanwhile, the feature point positioning precision is ensured through cascaded deep learning detection, and the precision and robustness of a sight line tracking system are remarkably improved.

The invention discloses an image rendering method and system, and the method comprises the following steps: collecting a fixation point of a user through eye movement tracking, and mapping the fixation point into a fixation position in a three-dimensional space; the system constructs a hierarchical sensing area model with the gazing position as the center, and rendering priorities of different areas are divided. A Gaussian kernel is generated after original three-dimensional point cloud data is subjected to structural analysis, and a weighted three-dimensional Gaussian point cloud data set is generated in combination with the priority of a sensing area. The system schedules Gaussian point cloud by using a spatial index structure, and dynamically generates pixel resolution, sampling step length and illumination parameters according to a gazing distance. And finally, differential rendering is performed on the target point set through a three-dimensional Gaussian sputtering technology, and a current frame image is generated. The method can be widely applied to real-time image rendering scenes needing high quality and high efficiency.

Method for designing a mechatronic component, characterized by the following features: The component is repeatedly simulated by software (10) with certain design parameters (11) that take on variable values, A fixation point (20) of the designer (30) is captured on a screen, while the software (10) displays the component to the designer (30) on the screen. A neural circuit with LTC neurons is trained to infer the resulting fixation point (20) of the designer (30) from the current values ​​of the design parameters (11), and The values ​​of the design parameters (11) are modified by the neural circuit and fed back to the software (10).

The invention relates to the technical field of augmented reality, in particular to a guide AR glasses interaction method and system based on gesture recognition, and the method comprises the steps: obtaining a control instruction triggered when a user eyeball fixation point is located in each page region in a historical period, analyzing the triggering frequency of the control instruction of each page region, and obtaining the control instruction of each page region; screening a control instruction corresponding to the page area, constructing an instruction triggering rule set exclusive to the page area, collecting a user image, identifying an age group and determining a dynamic inclusion coefficient; an eyeball fixation point is obtained in real time, a corresponding page area is marked, a demand control instruction is obtained from a corresponding instruction triggering rule set according to the marked page area, and a coverage area is determined based on a containing coefficient to generate an optimized gesture action sample; the determination is performed preferentially based on the optimized gesture action sample. Based on different functions of different areas of the page, the judgment standard is adaptively optimized, the accuracy of gesture judgment and recognition is improved, the response speed is increased, and the misjudgment rate is reduced.

The invention provides a fixation point tracking method and device based on a three-dimensional sight line, and relates to the field of intelligent perception. The method comprises the steps of performing simplified modeling on an eyeball model based on an eyeball vision structure, and obtaining key construction parameters in a three-dimensional space where a user is located according to the eyeball model; obtaining an eye image of the user, and constructing a three-dimensional pupil section parameter corresponding to the user based on the key construction parameter and by performing reverse geometric projection on an elliptical contour in the eye image; calculating eyeball center parameters and eyeball radius parameters based on the three-dimensional pupil section parameters; and determining gaze direction information of the user according to the eyeball center parameter and the eyeball radius parameter, and obtaining a gaze point corresponding to the user based on the fitting polynomial so as to track the gaze point of the user. According to the method and the device, the problem that when a user has a high-dynamic scene with a quick watching switching behavior, response lag still occurs to a sight line focus tracking effect, and the dynamic recognition precision is relatively low is solved.

The invention discloses a spatial orientation real-time analysis method based on eye movement, relates to the related technical field of flight training, and aims to overcome the defects of low analysis accuracy and poor adaptability in the prior art. The method comprises the following steps: acquiring pilot eye movement data through eye movement tracking equipment, acquiring head space orientation data through a flight helmet space sensor, and acquiring environment data containing instrument visual complexity and aircraft attitude from a virtual flight simulator; performing time synchronization on the three types of data; extracting features such as coordinates of a fixation point from the synchronous eye movement data; comparing the head orientation data with the reference data to generate deviation data; inputting the eye movement characteristics, the deviation data and the environment data into an evaluation model; the model dynamically adjusts parameters according to environment data, and calculates pilot space orientation capability indexes in combination with eye movement characteristics and deviation data. According to the method, accurate integration of multi-dimensional data can be realized, the analysis comprehensiveness and reliability are improved, a dynamic flight scene is adapted, and real-time accurate evaluation is realized.

The invention discloses a rendering method and device of wearable equipment, equipment and a storage medium, relates to the technical field of wearable equipment, and discloses the rendering method of the wearable equipment, the method is applied to the wearable equipment, an eye tracking device is arranged, and the eye tracking device is used for monitoring eye movement. The method comprises the following steps: analyzing an eye tracking image collected by an eye tracking device, and determining a gazing point location and gazing parameters; determining a device offset state according to the gaze parameter; and when the state is not shifted, determining a target rendering area according to the fixation point location and the equipment interaction information, and performing image enhancement rendering in the area. Through the above mode, whether deviation exists between the equipment and the eyes of the user is determined based on the gaze parameter, the rendering area is determined by combining the gaze point location and the equipment interaction information when deviation does not occur, rendering is enhanced in the area, visual delay and scene deviation are effectively reduced, motion sickness of the user is relieved, and user experience is improved.

The invention relates to an eye movement tracking transparent near-to-eye display optical structure based on deflectable pixel light, which relates to the field of photoelectric display and comprises a display module, a light deflection module and an eye movement tracking module, the display module comprises a transparent substrate, a light-emitting pixel array and a micro-collimation element. The light deflection module comprises a polarization element array wrapped by magnetic materials, voltage is applied to polarization elements to generate electromagnetic force to control the deflection angle of single pixel light, and the light is made to point to the center of the pupil of the human eye. The eye movement tracking module comprises a visual sensor and a microprocessor, the visual sensor is used for acquiring human eye fixation point information, the microprocessor controls the lightening state and the deflection angle of a corresponding pixel according to the fixation point information, and selective deflection and light emitting of the pixel in a fixation point area are achieved. The retina projection display is realized through pixel-level light deflection, the light is directly focused on the retina, and visual fatigue caused by depth mismatching of a virtual image and a real environment in traditional near-to-eye display is eliminated.

A testing system is provided with a computing device and a camera, the computing device operates the method for obtaining the eye fixation point, and the method comprises the following steps: firstly, shooting continuous frames of images through the camera, judging the position of a face frame by frame, obtaining the image characteristics of the face, obtaining the positions of two eyes, and obtaining the positions of the two eyes; and then calculating confidence degrees of a plurality of feature points of each eye by using a deep learning model, namely selecting one eye according to the total confidence degree of each eye, executing personalized correction to eliminate displacement between a fixation point formed by personalized differences and a target point at the moment, and then calculating a fixation direction of the eye. And determining an eye center point according to the gazing direction and the selected feature points of the eyes so as to judge a gazing point, and projecting the gazing point on a display screen.

The invention discloses a state sensing method, wearable equipment, a storage medium and a program product, and relates to the technical field of wearable equipment, the state sensing method is applied to the wearable equipment, and the method comprises the following steps: determining whether a user fixation point is in a content display range of the wearable equipment through a preset eye movement tracking module; under the condition that the user fixation point is in the content display range, monitoring the fixation time of the user fixation point; determining whether the attention of the user is concentrated in a content display range or not according to whether the watching time exceeds a preset time threshold value or not; and outputting external prompt information under the condition that the attention of the user is focused on the content display range. According to the method and the device, the user state is sensed in real time, and the external prompt information which can be sensed by surrounding people is output, so that the external interaction use convenience of the wearable equipment is improved.

The invention relates to the technical field of data processing, and relates to a trajectory conformity calculation method and system based on eye movement data, and the method comprises the steps: obtaining an eye movement trajectory data set and a target trajectory data set; calculating an average distance between adjacent fixation points in the eye movement trajectory data set to represent trajectory sparseness; determining a reference hit radius based on the size of a target object in the target trajectory data set, and multiplying the reference hit radius by a sparsity correction factor determined according to the trajectory sparsity to obtain a dynamic effective hit radius; calculating a plurality of conformity indexes between the eye movement track and the target track; according to the numerical value of each conformity index, calculating a dynamic weight corresponding to the conformity index; and performing weighted geometric average calculation on each conformity index and the corresponding dynamic weight to obtain a final track conformity score. According to the scheme of the invention, the authenticity, robustness and dynamic adaptability of track conformity evaluation are significantly improved.

This disclosure provides a screen display brightness adjustment method and apparatus, electronic device, and computer-readable storage medium, relating to the field of device control technology. The method includes: determining a salience parameter of a displayed image on a screen interface, the salience parameter describing the salience of each subject in the displayed image; determining a fixation point parameter where the current human eye's gaze falls on the current screen interface, the fixation point parameter representing the area of ​​the current human eye's gaze; determining brightness adjustment parameters for corresponding areas of each subject in the displayed image based on the salience parameter and the fixation point parameter; and adjusting the display brightness of each area on the screen interface according to the brightness adjustment parameters. This disclosure, by determining the salience of each subject in the displayed image and the area of ​​the human eye's gaze on the screen, and accordingly determining the brightness adjustment parameters, can adjust the brightness of the observed subject to a suitable brightness for the human eye while reducing the brightness of unnoticed subjects.

A device for evaluating the performance of visual equipment for a wearer. The device includes at least one input adapted to obtain virtual tests to be performed with the equipment, each test including at least one scenario combined with at least one virtual model of the wearer, defining how a visual task including a sequence of fixation points is to be carried out by the model in an environment defined by a description of shapes and positions of elements to be viewed by the model, at least one processor configured for selecting at least one test, based on at least one personalized real or simulated wearer activity profile representing usage of the equipment by the wearer, and evaluating the performance of the equipment with which the selected test is performed by the model by computing for one or more fixation points at least one performance criterion generated for the task.

The invention relates to the technical field of animation rendering, in particular to an animation scene fixation point rendering system based on eye movement tracking, which comprises a semantic attention modulation module, a rendering area dynamic construction module and a layered sample distribution rendering module, the semantic attention modulation module generates an attention diffusion diagram of semantic modulation based on the time sequence semantic importance maps of the current animation frame and the preorder frame and the current instantaneous fixation point data; a rendering area dynamic construction module adaptively constructs a core rendering area and a gradual transition rendering area according to the attention diffusion diagram; and the layered sample distribution rendering module implements ray tracing sampling of different sample numbers on the core rendering area, the gradual transition rendering area and the background area on the GPU. Compared with the prior art, the image quality of the gazing area is improved under the limited computing power, and the overall visual transition is kept smooth.

The present application relates to a kind of video perspective camera automatic focusing method and system for XR head-mounted display, belong to image processing technical field, solve the existing video perspective camera focusing speed slow, and the depth information of monocular image and focusing parameter cannot be effectively mapped problem.Method includes: the current image and its corresponding depth map of the video perspective camera of XR head-mounted display are collected;According to the fixation point coordinates of user in current image, the local region with fixation point coordinates as center is extracted from depth map as fixation area depth map;Fixation area depth map and fixation point coordinates are input into trained focusing parameter prediction model, and the target focusing parameter of video perspective camera is obtained;According to target focusing parameter, video perspective camera is adjusted, and single focusing is completed.Accurate mapping from image to focusing parameter is realized, and focusing speed is improved.

The present invention relates to a method for screening mobile terminal visual attention abnormalities in children based on multimodal data learning. A calibration video and a testing video are set up, and a head-face video of children while watching the calibration video and the testing video on smartphones is recorded, respectively. An eye-tracking estimation model is constructed to predict the fixation point location from the head-face video corresponding to the testing video frame by frame and to extract the eye-tracking features. Facial expression features and head posture features are extracted. A Long Short-Term Memory (LSTM) network is used to fuse different modal features and realize the mapping from multimodal features to category labels. In the testing stage, the head-face video of children to be classified while watching the videos on smartphones is recorded, and the features are extracted and input into the post-training model to determine whether they are abnormal.

The invention relates to an artificial intelligence-based precise eye movement tracking identification system and method. The intelligent interaction tracking expansion module is used for integrating and packaging a multi-source database into an HTML (Hypertext Markup Language) instruction file set, constructing a video eye movement tracking data collection platform, and intelligently interacting and expanding eye movement tracking information and high-precision eye movement tracking; the eye movement tracking and calibration module is used for controlling eye movement tracking and triggering calibration, and controlling camera access, eye movement key point identification, fixation point prediction calibration and precision test; the video synchronization and eye movement analysis module is used for constructing an immersive video playing environment, associating eye movement data with a video timestamp and carrying out gaze point eye jump behavior recognition statistics; and the data visualization and cloud storage module performs thermodynamic diagram generation rendering, composite screen snapshot generation, eye movement structured data generation and composite screen snapshot cloud storage, monitors a database loading state, a database connection state and an eye movement data uploading result, and tracks the reliability of a data acquisition process.

The invention discloses a non-inductive three-dimensional scene interest point marking method based on intention recognition and a terminal. The method comprises the following steps: S1, presenting a three-dimensional scene; s2, detecting a gazing behavior of a user, judging whether an effective gazing event exists or not, and if the effective gazing event is detected, entering a step S3; s3, recording the spatial position coordinates of the fixation point on the three-dimensional model; s4, extracting an electroencephalogram signal fragment locked with the time of the watching event by using an FRP brain-computer interface module; s5, judging whether an FRP signal representing the marking intention exists or not; s6, fusing the spatial position coordinates and the intention confirmation result to generate a final marking instruction; and S7, placing a virtual marker at the position coordinate specified by the marker according to the marking instruction. According to the invention, high-precision and non-inductive remote marking operation capable of liberating two hands can be realized, and the interaction efficiency and accuracy of remote operation in a high-risk environment are remarkably improved.