Hybrid wireless eye tracking method and system based on multi-sensor fusion, electronic device, and storage medium

Abstract

A hybrid wireless eye tracking method and system based on multi-sensor fusion, an electronic device, and a storage medium, relating to the technical field of eye tracking. The method comprises: first using a plurality of different types of sensors to acquire a plurality of types of data; then preprocessing the plurality of types of sensor data; subsequently fusing the preprocessed sensor data on the basis of a graph-theoretic method; and finally performing gaze trajectory prediction on the basis of a convolutional neural network and a gated recurrent unit.

Description

Hybrid wireless eye-tracking method, system, electronic device, and storage medium based on multi-sensor fusion

[0001] Technical Field

[0002] This invention relates to the field of eye-tracking technology, specifically to a hybrid wireless eye-tracking method, system, electronic device, and storage medium based on multi-sensor fusion.

[0003] Background Technology

[0004] Eye-tracking technology, as a key technique for exploring visual and human behavior across multiple fields, is increasingly demonstrating its importance. By constructing more natural and intuitive gaze interfaces and providing precise analysis, eye-tracking technology has significantly advanced many fields, including human-computer interaction, virtual and augmented reality, medical diagnosis, and psychological research. With the continuous development and evolution of eye-tracking technology, the demand for new systems adapted to different usage scenarios is becoming increasingly urgent. These fields and scenarios require systems with accurate and robust tracking performance, high-speed data transmission capabilities, energy efficiency, and the ability to achieve more natural and intuitive human-computer interaction.

[0005] However, traditional eye-tracking systems generally rely on single-modal sensors, which inherently limits their tracking accuracy and stability. These limitations are particularly pronounced in dynamic environments or when head movement processing is involved. Furthermore, traditional eye-tracking systems also have significant shortcomings in key areas such as real-time wireless data transmission efficiency and power efficiency.

[0006] In view of this, this application proposes a hybrid wireless eye-tracking technology solution based on multi-sensor fusion, which aims to solve at least one or more of the aforementioned constraints, so as to optimize and break through existing eye-tracking technologies.

[0007] Summary of the Invention

[0008] (a) Technical problems to be solved

[0009] To address the shortcomings of existing technologies, this invention provides a hybrid wireless eye-tracking method, system, electronic device, and storage medium based on multi-sensor fusion, which at least solves the technical problems of poor tracking accuracy and poor stability in existing eye-tracking technologies.

[0010] (II) Technical Solution

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] Firstly, this application proposes a hybrid wireless eye-tracking method based on multi-sensor fusion, the method comprising:

[0013] It uses a multi-sensor system, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera, to collect comprehensive data on the user's eye movement and head position.

[0014] Perform data fusion on the comprehensive data;

[0015] The user's gaze direction is estimated based on the fused sensor data.

[0016] In one embodiment, the integrated data is preprocessed before data fusion; the preprocessing includes:

[0017] Perform graphical Fourier transform on the data acquired by the Q-Var sensor; and / or

[0018] Perform Kalman filtering on the data acquired by the inertial measurement unit; and / or

[0019] Motion blur compensation is applied to the data collected by the high-speed panoramic camera.

[0020] Preferably, the preprocessing further includes:

[0021] Adaptive thresholding is performed on Q-Var sensor data after graphical Fourier transform based on spectral clustering technology.

[0022] In one embodiment, data fusion of the comprehensive data includes:

[0023] A graph theory-based method is used to fuse preprocessed data from Q-Var sensors, inertial measurement units, and high-speed panoramic cameras.

[0024] In one embodiment, the method further includes: extracting key eye-tracking features from the fused data.

[0025] In one embodiment, estimating the user's gaze direction based on the fused sensor data includes:

[0026] User gaze trajectory prediction is performed based on the analysis and fusion of sensor data using convolutional neural networks and gated recurrent units.

[0027] In one embodiment, the method further includes:

[0028] Use at least one of 5G millimeter wave technology or the IEEE 802.11ac protocol to wirelessly transmit gaze tracking data.

[0029] Preferably, the wireless transmission method is configured for low power consumption.

[0030] In one embodiment, the method further includes:

[0031] Set up an application layer that integrates edge computing capabilities to enable a variety of real-time eye-tracking applications.

[0032] In one embodiment, the method further includes:

[0033] The user interface is dynamically adapted based on the estimated gaze direction of the user.

[0034] Preferably, a personalized eye comfort mode with adaptive features is achieved based on game theory.

[0035] Secondly, this application further proposes a hybrid wireless eye-tracking system based on multi-sensor fusion, the system comprising:

[0036] The data acquisition module is configured to acquire comprehensive data on the user's eye movements and head position based on multiple sensors, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera.

[0037] The data fusion module is configured to perform data fusion on the comprehensive data;

[0038] The gaze estimation module is configured to estimate the user's gaze direction based on the fused sensor data.

[0039] When the data acquisition module, data fusion module, gaze estimation module and their associated modules execute the program, they implement the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described above.

[0040] Thirdly, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any of the preceding claims.

[0041] Fourthly, this application finally proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any of the preceding claims.

[0042] (III) Beneficial Effects

[0043] This invention provides a hybrid wireless eye-tracking method, system, electronic device, and storage medium based on multi-sensor fusion. Compared with existing technologies, it has the following advantages:

[0044] This application proposes a hybrid wireless eye-tracking method based on multi-sensor fusion. First, it acquires comprehensive data on the user's eye movements and head position using multiple sensors, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera. Then, it fuses this comprehensive data. Finally, it estimates the user's gaze direction based on the fused sensor data. This hybrid wireless eye-tracking method based on multi-sensor fusion improves the performance of eye-tracking technology under different environmental conditions, resulting in higher eye-tracking accuracy and better stability.

[0045] Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 is a flowchart of a hybrid wireless eye-tracking method based on multi-sensor fusion proposed in an embodiment of this application;

[0048] Figure 2 is a flowchart of a hybrid wireless eye-tracking method based on multi-sensor fusion proposed in another embodiment of this application;

[0049] Figure 3 is a flowchart of the Q-Var sensor data acquisition, processing and transmission process in an embodiment of this application;

[0050] Figure 4 is a flowchart of Q-Var sensor data preprocessing in an embodiment of this application.

[0051] Figure 5 is a flowchart of a hybrid wireless eye-tracking method based on multi-sensor fusion proposed in another embodiment of this application;

[0052] Figure 6 is a logic diagram of a hybrid wireless eye-tracking method based on multi-sensor fusion proposed in another embodiment of this application;

[0053] Figure 7 is a schematic diagram of the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in an embodiment of this application;

[0054] Figure 8 is a schematic diagram of the principle of the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in the embodiments of this application.

[0055] Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Definitions:

[0058] Eye tracking: Eye tracking technology is a non-invasive technique that records what a subject is looking at, primarily based on eye video analysis. This technology uses a near-infrared light source to generate reflected images on the user's cornea and pupil. Two image sensors then capture these images of the eyes and their reflections. Image processing algorithms and a three-dimensional eye model are used to precisely calculate the eye's spatial position and gaze location. Eye tracking technology has become a key tool in many fields related to visual behavior and human behavior, such as psychology, neuromarketing, neurocognition, user experience, basic research, and market research.

[0059] Inertial Measurement Unit (IMU): An inertial measurement unit (IMU) is a device used to measure the motion state of an object. It typically consists of sensors such as accelerometers, gyroscopes, and magnetometers, and can measure information such as the object's acceleration, angular velocity, and magnetic field strength to determine the object's attitude, position, and velocity.

[0060] High-speed panoramic camera: A high-speed panoramic camera is a camera device that can quickly capture high-resolution panoramic images. It features high resolution, fast shooting, panoramic coverage, real-time transmission and processing, and is widely used in security monitoring, sports broadcasting, autonomous driving, virtual reality (VR) and augmented reality (AR), industrial inspection and other fields.

[0061] Motion blur compensation: Motion blur is an image blurring phenomenon caused by the relative movement of the camera or the subject during exposure. Motion blur compensation uses various techniques and methods to reduce or eliminate this blur, thereby improving image sharpness and quality.

[0062] This application provides a hybrid wireless eye-tracking method, system, electronic device, and storage medium based on multi-sensor fusion, which at least solves the technical problems of poor tracking accuracy and stability in existing eye-tracking technologies, and achieves the goals of high accuracy, high stability, high efficiency, energy saving, and intelligence in user gaze estimation.

[0063] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0064] To address the challenges of accuracy, real-time performance, and power efficiency often faced by traditional eye-tracking technologies, particularly poor accuracy and stability in dynamic environments or when processing head movements, this application combines multiple sensor modalities and advanced processing techniques to overcome these limitations and push the boundaries of eye-tracking technology. The main technologies include: combining multiple sensor types such as Q-Var sensors, inertial measurement units (IMUs), and high-speed panoramic cameras to improve gaze tracking accuracy, head motion compensation, and real-time data transmission. To enhance gaze estimation accuracy and stability, data preprocessing techniques including Graph Fourier Transform (GFT), Kalman filtering, and motion blur compensation, as well as graph theory-based sensor fusion, are used to process and fuse multi-sensor data. Gaze trajectory estimation is achieved based on convolutional neural networks (CNNs) and gated recurrent networks (GRUs). For data transmission, this application also incorporates 5G mmwave and IEEE 802.11ac wireless transmission to enhance bandwidth and low-latency data transmission, providing energy-efficient, real-time gaze estimation under various environmental conditions. This application also features a game theory model for gaze adaptation and improved user interaction in dynamic environments, as well as adaptive personalized eye comfort modes.

[0065] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0066] Example 1:

[0067] Firstly, this invention proposes a hybrid wireless eye-tracking method based on multi-sensor fusion, as shown in Figure 1. This method includes:

[0068] It uses a multi-sensor system, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera, to collect comprehensive data on the user's eye movement and head position.

[0069] Perform data fusion on the comprehensive data;

[0070] The user's gaze direction is estimated based on the fused sensor data.

[0071] This embodiment proposes a hybrid wireless eye-tracking method based on multi-sensor fusion. First, it collects comprehensive data on the user's eye movements and head position using multiple sensors, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera. Then, it fuses this comprehensive data. Finally, it estimates the user's gaze direction based on the fused sensor data. This method improves eye-tracking performance under different environmental conditions, achieving higher accuracy and better stability.

[0072] The implementation process of an embodiment of the present invention will be described in detail below with reference to the explanation of the specific steps described above.

[0073] This embodiment proposes a hybrid wireless eye-tracking method based on multi-sensor fusion, as shown in Figures 1-4 and 6. Its implementation steps mainly include:

[0074] S1. Collect comprehensive data on user eye movement and head position based on multiple sensors including Q-Var sensors, inertial measurement units, and high-speed panoramic cameras.

[0075] Traditional eye-tracking technologies typically rely on single-modal data acquired by a single-modal sensor, which can limit the accuracy of user annotation estimation, especially in dynamic environments or when processing head movements. Therefore, this embodiment proposes a hybrid wireless eye-tracking method based on multi-sensor fusion, utilizing a combination of sensors including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera to acquire comprehensive data on user eye movements and head position. Specifically:

[0076] First, the user's electrooculogram (EOG) signal is captured using a Q-Var sensor. The Q-Var sensor can detect minute potential differences generated by eye movements. The EOG signal captured by the Q-Var sensor can provide information about eye position and movement with high temporal resolution.

[0077] Then, an inertial measurement unit (IMU) including at least a gyroscope and an accelerometer is used to collect the user's head motion data. The gyroscope is used to measure the changes in the angular velocity and direction of the user's head, and the accelerometer is used to detect linear acceleration and tilt.

[0078] In a preferred embodiment, the inertial measurement unit (IMU) further includes a magnetometer for magnetometer calibration of the IMU data. The magnetometer calibration process involves compensating for hard and soft iron effects that can distort magnetometer readings. Proper magnetometer calibration can improve the accuracy of heading information provided by the IMU's magnetometer assembly, thereby further ensuring the accuracy of head position and orientation estimations from the IMU data.

[0079] When collecting user head motion data, IMU components such as gyroscopes, accelerometers, and magnetometers work together to more accurately track the user's head movement and position.

[0080] Finally, a high-speed panoramic camera captures images of the user's surroundings in front of them. This high-speed panoramic camera can provide contextual information, allowing the user to associate eye movements with objects or areas of interest within their field of vision.

[0081] Of course, in practical applications, other types of sensors can be added as needed. Therefore, the aforementioned multiple sensors include, but are not limited to, Q-Var sensors, inertial measurement units, and high-speed panoramic cameras. The comprehensive data collected in conjunction with these sensors includes, but is not limited to, user eye movement and head position data, as well as data from other parts or organs of the user's head, whose movements are related to the user's gaze.

[0082] As can be seen, compared with existing single-sensor solutions, the integration of multiple sensors, including the Q-Var sensor, IMU, and high-speed panoramic camera in this embodiment, provides more robust and accurate raw data for gaze tracking estimation, which helps to improve the accuracy of subsequent gaze estimation. Moreover, this method of acquiring comprehensive data from multiple sensors allows for better compensation of head movements and improves performance under different environmental conditions, resulting in better stability of gaze estimation.

[0083] S2. Perform data fusion on the comprehensive data.

[0084] After acquiring comprehensive data on user eye movement and head position using multiple sensors, including Q-Var sensors, inertial measurement units (IMUs), and high-speed panoramic cameras, the data from these sensors is fused. This data fusion process leverages the strengths of each sensor mode to produce a more robust and accurate representation of eye movement.

[0085] In one embodiment, a graph-based approach is utilized to combine data from a Q-Var sensor, an IMU, and a high-speed panoramic camera. This sensor data fusion process involves constructing a graph representation of the multimodal data, where nodes represent different sensor inputs and edges represent relationships between these inputs. Graph-based sensor data fusion methods can more effectively integrate heterogeneous sensor data, potentially improving the overall accuracy of gaze estimation.

[0086] In a preferred embodiment, referring to Figures 2-4, before performing data fusion on the integrated data, the above-mentioned hybrid wireless eye-tracking method based on multi-sensor fusion further includes:

[0087] S20. Preprocess the above-mentioned comprehensive data; the preprocessing includes:

[0088] Perform graphical Fourier transform on the data acquired by the Q-Var sensor; and / or

[0089] Perform Kalman filtering on the data acquired by the inertial measurement unit; and / or

[0090] Motion blur compensation is applied to the data collected by the high-speed panoramic camera.

[0091] In practical implementation, based on the actual needs for accuracy and data processing efficiency, preprocessing can be performed on one or more of the aforementioned sensor data types. For example, for Q-Var sensor data, graph Fourier transform and spectral clustering techniques can be used for preprocessing; and / or Kalman filter filtering and magnetometer calibration can be used to preprocess IMU data; and / or motion blur compensation processing can be performed on the information stream from the high-speed panoramic camera. Specifically:

[0092] Graphical Fourier Transform (GFT) preprocessing is performed on Q-Var sensor data to transform the raw sensor data into a form suitable for accurate gaze estimation, as shown in Figures 3-4. The GFT component can be applied to Q-Var sensor data to convert the time-domain signal of the Q-Var sensor data into the graphical frequency domain. This transformation allows low-frequency gaze points to be separated from high-frequency noise and microsaccades. GFT processing of Q-Var sensor data includes: constructing a graphical representation of the Q-Var data, calculating the Laplacian quantity of the graph, and applying a transform to convert the signal to the spectral domain.

[0093] For IMU data, a Kalman filter is used to perform Kalman filtering. The Kalman filter can be used to estimate the true state of the IMU sensor by combining predictions based on previous states with new measurements. Kalman filtering helps reduce noise and improves the accuracy of head position and orientation estimation from IMU data.

[0094] Data acquired by high-speed panoramic cameras is processed using motion blur compensation technology. Motion blur compensation analyzes the information stream from the high-speed panoramic camera to detect and correct blur caused by rapid head or eye movements. Motion blur compensation helps maintain a sharp image from the high-speed panoramic camera and is important for associating the user's gaze direction with environmental features.

[0095] In a more preferred embodiment, the preprocessing of the Q-Var sensor data, after the GFT, includes further processing using spectral clustering techniques applied to the Q-Var data after the graph Fourier transform. Spectral clustering can classify different eye movement events, such as fixation, saccades, and blinks, through adaptive thresholding. Spectral clustering can utilize the frequency representation of the graph to identify different clusters corresponding to different eye movement states.

[0096] The data preprocessing steps in the above embodiments of this application can preprocess one or more types of sensor data, preparing higher-quality data for subsequent sensor data fusion and gaze estimation processes. By processing sensor-specific noise and artifacts during the data preprocessing stage, the overall accuracy and robustness of the hybrid wireless eye-tracking system proposed in this application are improved.

[0097] In a further preferred embodiment, referring to Figures 5-6, the hybrid wireless eye-tracking method based on multi-sensor fusion proposed in the above embodiments further includes feature extraction of the fused data, and analysis of the preprocessed sensor data to extract key eye-tracking features.

[0098] Feature extraction can receive input from the preprocessing stages of the Q-Var sensor and IMU. During feature extraction, preprocessed Q-Var sensor data can be analyzed to determine pupil size. The electrical signals captured by the Q-Var sensor may be correlated with changes in pupil diameter; by examining patterns and amplitudes in the processed Q-Var data, the feature extraction module can estimate the relative pupil size over time. Feature extraction can also extract information about fixation duration from the sensor data. In some cases, the relative stable periods of the preprocessed Q-Var and IMU signals are analyzed. This feature extraction step can identify sustained periods where the eye position remains relatively constant, indicating visual fixation on a specific point or object. Furthermore, feature extraction can process sensor data to detect and quantify microsaccades. Microsaccades are small, rapid eye movements that occur during visual fixation. In some cases, this feature extraction approach can apply frequency analysis techniques to the preprocessed Q-Var sensor data to identify these high-frequency, low-amplitude eye movements. This feature extraction can track the frequency and characteristics of detected microsaccades over time.

[0099] Furthermore, the graphical Fourier transform applied during Q-Var sensor preprocessing can also facilitate feature extraction. In some cases, the transformed signal can allow for more effective separation of different eye movement components. Graph Fourier transform can enable adaptive thresholding of Q-Var sensor data, thereby improving the accuracy of feature extraction. Spectral clustering techniques can be applied to the transformed Q-Var data for adaptive thresholding. In some cases, this method may help distinguish different eye movement states and events. Spectral clustering can enable the feature extraction module to more accurately classify fixation periods, saccades, and other eye movement periods.

[0100] By extracting these key features—pupil size, fixation time, and microsaccade frequency—from preprocessed sensor data, rich eye movement characteristics can be provided. This extracted information can then serve as input data for subsequent fixation estimation and analysis processes in hybrid wireless eye-tracking systems.

[0101] S3. Estimate the user's gaze direction based on the fused sensor data.

[0102] The gaze direction is determined by combining the fused data or the features extracted from the fused data with the data collected by the high-speed panoramic camera after motion blur compensation.

[0103] In a preferred embodiment, a convolutional neural network (CNN) and a gated recurrent unit (GRU) are used for gaze trajectory prediction when estimating the user's gaze direction. The CNN component can analyze spatial features extracted from sensor data, while the GRU can process time series data to predict gaze trajectories that change over time.

[0104] The gaze estimation process can combine features from fused data or features extracted from fused data, including information about pupil size, gaze duration, and microsaccade frequency. Furthermore, the fused data or features extracted from fused data can also include motion-compensated data from a high-speed panoramic camera, providing environmental context for gaze estimation.

[0105] In a more preferred embodiment, during gaze estimation, the CNN applies multiple convolutional layers to extract features from the fused data or features extracted from the fused data, as well as relevant features extracted from the input data of the motion blur compensation unit. These convolutional layers can detect patterns and features in the spatial domain that indicate the gaze direction.

[0106] In a further embodiment, the GRU component for gaze estimation can process feature sequences over time. In practice, the GRU maintains an internal state that allows it to capture temporal dependencies in eye movement patterns. This cyclical structure may enable the module to predict future gaze trajectories based on past and current inputs.

[0107] Gaze estimation can fuse the outputs from CNN and GRU components to produce a final estimate of the gaze direction. In practice, the fusion process involves a weighted combination of spatial and temporal features to produce a comprehensive gaze prediction.

[0108] Gaze estimation can also incorporate adaptive mechanisms to improve its gaze estimation accuracy over time. In one embodiment, a hybrid wireless eye-tracking system based on multi-sensor fusion can use feedback from a high-speed panoramic camera or other sensors to calibrate and improve the gaze estimation model during use.

[0109] By leveraging the combined capabilities of spatial analysis using CNNs and temporal processing using GRUs, gaze estimation can provide accurate and robust gaze direction prediction. This gaze estimation method enables hybrid wireless eye-tracking systems to handle complex eye movement patterns and adapt to various user behaviors and environmental conditions.

[0110] In a further embodiment, referring to Figures 6 and 8, the above-described hybrid wireless eye-tracking method based on multi-sensor fusion further includes:

[0111] Use at least one of 5G millimeter wave technology or the IEEE 802.11ac protocol to wirelessly transmit gaze tracking data.

[0112] The 802.11ac module for wireless transmission can be configured to use multiple wireless protocols to transmit data. These wireless transmission methods include, but are not limited to, using at least one of 5G millimeter-wave technology or the IEEE 802.11ac protocol to wirelessly transmit gaze tracking data.

[0113] In one embodiment, wireless transmission can utilize 5G millimeter wave (mmWave) technology for data transmission. 5G mmWave technology operates at high frequencies (between 30 GHz and 300 GHz). Compared to low-frequency wireless technologies, this high-frequency operation allows for increased bandwidth and reduced latency.

[0114] In another preferred embodiment, the wireless transmission method further includes the IEEE 802.11ac protocol for data transmission. In some implementations, the 802.11ac protocol can operate in the 5 GHz band and provide high-throughput wireless communication.

[0115] By combining 5G mmWave and IEEE 802.11ac technologies, wireless transmission methods can offer greater flexibility in data transmission. In practice, dynamic switching between these methods can be achieved based on factors such as signal strength, bandwidth requirements, and power consumption. Dual transmission capabilities can provide advantages in various use cases. For example, 5G mmWave technology may be suitable for applications requiring extremely low latency and high bandwidth, such as real-time virtual reality. Conversely, IEEE 802.11ac may be more suitable for indoor environments where mmWave signals may have difficulty penetrating walls.

[0116] In a more preferred embodiment, the wireless transmission method can be configured for low-power operation. In this case, the wireless transmission method is based on techniques such as adaptive power control, where the transmission power is adjusted according to signal quality and distance to the receiver. The wireless transmission method can also implement energy-saving protocols to reduce energy consumption during periods of inactivity.

[0117] Wireless transmission methods can support real-time data transmission of gaze tracking information. In some cases, time-sensitive data packets can be prioritized, and Quality of Service (QoS) mechanisms can be implemented to ensure consistent, low-latency transmission of critical eye-tracking data.

[0118] By leveraging the advantages of 5G mmWave and IEEE 802.11ac technologies, wireless transmission methods can provide robust, flexible, and efficient data transmission capabilities for hybrid wireless eye-tracking systems. This dual-mode approach enables the system to adapt to various environmental conditions and application requirements while maintaining high performance and energy efficiency.

[0119] In one embodiment, the above-mentioned hybrid wireless eye-tracking method based on multi-sensor fusion further includes:

[0120] Set up an application layer that integrates edge computing capabilities to enable a variety of real-time eye-tracking applications.

[0121] As shown in Figures 6 and 8, the application layer can receive eye-tracking data transmitted wirelessly. In some cases, the application layer can implement dynamic user interface adaptation based on real-time gaze behavior. Hybrid wireless eye-tracking methods based on multi-sensor fusion can analyze gaze tracking data to determine the user's viewing location on the display or in the environment. Based on this analysis, the application layer can dynamically adjust user interface elements to optimize the user experience. For example, the application layer can zoom in or highlight the interface element the user is currently viewing, or move important information to areas where the user's gaze is frequently directed. In some implementations, the system can use predictive algorithms to predict the user's next likely viewing location and preload content in those areas for faster response times.

[0122] In one embodiment, the hybrid wireless eye-tracking method based on multi-sensor fusion further includes: achieving dynamic user interface adaptation based on the estimated gaze direction of the user.

[0123] The application layer can receive transmitted user gaze tracking data and utilize edge computing resources from various applications to achieve adaptive, personalized eye comfort modes.

[0124] In a preferred embodiment, a personalized eye comfort mode with adaptive features is implemented based on game theory.

[0125] In some cases, the application layer can use evolutionary game theory to implement adaptive, personalized eye comfort modes. These modes dynamically adjust display parameters or content presentation based on the user's gaze patterns and environmental conditions. The evolutionary game theory approach to eye comfort modes might involve treating different comfort-related parameters as strategies in a game. These strategies can be iteratively updated based on user feedback and gaze behavior, evolving towards the optimal comfort configuration over time. This adaptive mechanism could allow the system to provide a personalized eye-tracking experience for individual users, while taking into account constantly changing environmental factors.

[0126] The hybrid wireless eye-tracking method based on multi-sensor fusion proposed in this embodiment can be implemented in various fields requiring accurate eye tracking and gaze estimation, such as augmented reality (AR) and virtual reality (VR) applications. By providing more accurate and sensitive eye tracking, it can enhance immersion and user experience in AR / VR environments, enabling more natural interaction with virtual objects and improving the overall performance of these technologies.

[0127] In the medical field, the hybrid wireless eye-tracking method based on multi-sensor fusion proposed in this embodiment can significantly improve the diagnostic capabilities for neurological and ophthalmic diseases. Its high-precision gaze tracking and real-time data transmission characteristics make it particularly suitable for the early detection of eye movement abnormalities associated with diseases such as Parkinson's disease, multiple sclerosis, or certain types of brain tumors.

[0128] With potential applications in the automotive industry, particularly in advanced driver assistance systems (ADAS) and autonomous vehicles, the hybrid wireless eye-tracking method based on multi-sensor fusion proposed in this embodiment can enhance safety features, improve driver monitoring systems, and contribute to the development of more sophisticated vehicle human-machine interfaces by accurately tracking the driver's gaze and attention.

[0129] In the field of human-computer interaction, the hybrid wireless eye-tracking method based on multi-sensor fusion proposed in this embodiment can fundamentally change the way users interact with various devices and interfaces. By combining accurate gaze estimation with an adaptive user interface based on game theory models, it can bring more intuitive and efficient interaction methods to computers, smartphones, and other electronic devices, potentially replacing or supplementing traditional input methods such as mice and keyboards.

[0130] Example 2:

[0131] Secondly, the present invention also provides a hybrid wireless eye-tracking system based on multi-sensor fusion, as shown in Figure 7. This system mainly includes:

[0132] The data acquisition module is configured to acquire comprehensive data on the user's eye movements and head position based on multiple sensors, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera.

[0133] The data fusion module is configured to perform data fusion on the comprehensive data;

[0134] The gaze estimation module is configured to estimate the user's gaze direction based on the fused sensor data.

[0135] The implementation process of an embodiment of the present invention will be described in detail below with reference to Figures 6-8, the functions of the specific modules of the system proposed in this embodiment, and the specific connection relationships between the modules.

[0136] Figure 7 illustrates the principle block diagram of a hybrid wireless eye-tracking system architecture based on multi-sensor fusion. This hybrid wireless eye-tracking system combines multiple sensor technologies and advanced data processing techniques to provide accurate gaze tracking in a wireless configuration. Specifically:

[0137] The system includes a data acquisition module, which comprises at least a Q-Var sensor for capturing electrooculogram (EOG) signals, an inertial measurement unit (IMU), and a high-speed panoramic camera. These various types of sensors work together to collect comprehensive data on eye movements and head position from different angles, providing a rich dataset for accurate gaze tracking.

[0138] The system also includes a data fusion module configured to fuse data from the Q-Var sensor, IMU, and high-speed panoramic camera. During data fusion, the module leverages the strengths of each sensor mode to produce a more robust and accurate representation of eye movements.

[0139] The system further includes a gaze estimation module configured to analyze the fused sensor data to determine the user's gaze direction. The gaze estimation process utilizes machine learning techniques to interpret the complex relationships between various sensor inputs.

[0140] In one embodiment, the data acquisition module includes a Q-Var sensor for capturing electrooculogram (EOG) signals. The Q-Var sensor can detect minute potential differences generated by eye movements. The EOG signals captured by the Q-Var sensor can provide information about eye position and movement with high temporal resolution.

[0141] In one embodiment, the inertial measurement unit (IMU) includes a gyroscope and an accelerometer. The gyroscope is used to measure the angular velocity and orientation changes of the user's head, while the accelerometer is used for linear acceleration and tilt.

[0142] In a preferred embodiment, the inertial measurement unit (IMU) further includes a magnetometer for magnetometer calibration of the IMU data. The magnetometer calibration process involves compensating for hard and soft iron effects that can distort magnetometer readings. Proper magnetometer calibration can improve the accuracy of heading information provided by the IMU's magnetometer assembly, thereby further ensuring the accuracy of head position and orientation estimations from the IMU data.

[0143] The aforementioned IMU components, such as gyroscopes, accelerometers, and magnetometers, work together to more accurately track the movement and position of the user's head.

[0144] The high-speed panoramic camera in the data acquisition module can capture images of the user's surroundings in front of them. The high-speed panoramic camera can provide contextual information to correlate eye movements with objects or areas of interest within the user's field of vision.

[0145] The Q-Var sensor, IMU, and high-speed panoramic camera work together to collect diverse data as input sets. The Q-Var sensor can focus on eye-specific movements, while the IMU can track broader head movements, and the high-speed panoramic camera adds environmental context to the sensor data.

[0146] By combining these different sensor modalities, the data acquisition module can capture a comprehensive picture of eye and head dynamics. This approach, which uses multiple types of sensors to acquire different data from different angles and levels, allows the system to collect complementary data streams for fusion and analysis to produce accurate gaze tracking results.

[0147] In one embodiment, the aforementioned hybrid wireless eye-tracking system based on multi-sensor fusion further includes a data fusion module that fuses data from a Q-Var sensor, an IMU, and a high-speed panoramic camera. During data fusion, the data fusion module leverages the strengths of each sensor mode to produce a more robust and accurate representation of eye movements.

[0148] In a preferred embodiment, the data fusion module integrates algorithms and procedures including graph theory-based methods to fuse data from Q-Var sensors, IMUs, and high-speed panoramic cameras.

[0149] In one embodiment, the aforementioned hybrid wireless eye-tracking system based on multi-sensor fusion further includes a data preprocessing module. This module is configured to preprocess the raw data acquired from the different sensors before the multi-sensor data is fused. Specifically:

[0150] In one embodiment, the data preprocessing module includes separate components for processing data from one or more sensor types; that is, preprocessing can be performed on one or more types of sensor data as needed. For example, for Q-Var sensor data, preprocessing can be performed using graphical Fourier transform and spectral clustering techniques; and / or preprocessing can be performed using Kalman filter filtering and magnetometer calibration on magnetometer data; and / or motion blur compensation can be applied to the information stream from the high-speed panoramic camera. Specifically,

[0151] For Q-Var sensor data, the data preprocessing module includes a Graph Fourier Transform (GFT) component for preprocessing. The GFT component can be applied to the Q-Var sensor data, converting the time-domain signal of the Q-Var sensor data into the graph frequency domain. This conversion allows low-frequency gaze points to be separated from high-frequency noise and microsaccades. GFT processing of Q-Var sensor data includes constructing a graph representation of the Q-Var data, calculating the Laplace transform of the graph, and applying the transform to convert the signal to the spectral domain.

[0152] For IMU data, the data preprocessing module includes a Kalman filter to perform Kalman filtering on the IMU data. The Kalman filter can be used to estimate the true state of the IMU sensor by combining predictions based on previous states with new measurements. Kalman filtering helps reduce noise and improves the accuracy of head position and orientation estimation from the IMU data.

[0153] For data acquired by high-speed panoramic cameras, the data preprocessing module also includes a motion blur compensation unit. This unit is configured to process the data acquired by the high-speed panoramic cameras using motion blur compensation technology. Motion blur compensation involves analyzing the information stream from the high-speed panoramic camera to detect and correct blur caused by rapid head or eye movements. Motion blur compensation helps maintain a sharp image from the high-speed panoramic camera and is important for associating the user's gaze direction with environmental features.

[0154] In a preferred embodiment, the preprocessing of Q-Var sensor data, following the GFT, utilizes spectral clustering technology for further processing of the Q-Var data after the graph Fourier transform. Spectral clustering can classify different eye movement events, such as fixation, saccades, and blinks, through adaptive thresholding. Spectral clustering can identify different clusters corresponding to different eye movement states using the frequency representation of the graph.

[0155] The data preprocessing module proposed in the above embodiments of this application, through preprocessing steps for data from one or more sensor types, can prepare data for subsequent sensor fusion and gaze estimation processes. By processing sensor-specific noise and artifacts at this stage, the data preprocessing module contributes to the overall accuracy and robustness of the hybrid wireless eye-tracking system proposed in this application.

[0156] In a further preferred embodiment, the hybrid wireless eye-tracking system proposed in the above embodiments further includes a feature extraction module configured to process preprocessed and fused sensor data to extract key eye-tracking features. As shown in Figures 6 and 8, the feature extraction module can receive input from the preprocessing stages of the Q-Var sensor and the IMU.

[0157] The feature extraction module analyzes preprocessed Q-Var sensor data to determine pupil size. The electrical signals captured by the Q-Var sensor may be related to changes in pupil diameter. By examining the patterns and amplitudes in the processed Q-Var data, the feature extraction module can estimate the relative pupil size over time.

[0158] The feature extraction module can also extract information about gaze duration from sensor data. In some cases, it analyzes the relatively stable periods of preprocessed Q-Var and IMU signals. This feature extraction module can identify sustained periods in which the eye position remains relatively constant, indicating visual fixation on a specific point or object.

[0159] Furthermore, the feature extraction module can process sensor data to detect and quantify microsaccades. Microsaccades are small, rapid eye movements that occur during visual fixation. In some cases, the feature extraction module can apply frequency analysis techniques to preprocessed Q-Var sensor data to identify these high-frequency, low-amplitude eye movements. The feature extraction module can track the frequency and characteristics of detected microsaccades over time.

[0160] Furthermore, the graphical Fourier transform applied during Q-Var sensor preprocessing can also facilitate feature extraction. In some cases, the transformed signal can allow for more effective separation of different eye movement components. Graph Fourier transform can enable adaptive thresholding of Q-Var sensor data, thereby improving the accuracy of feature extraction. Spectral clustering techniques can be applied to the transformed Q-Var data for adaptive thresholding. In some cases, this method may help distinguish different eye movement states and events. Spectral clustering can enable the feature extraction module to more accurately classify fixation periods, saccades, and other eye movement periods.

[0161] By extracting key features—pupil size, fixation time, and microsaccade frequency—from preprocessed sensor data using a feature extraction module, the module can provide rich eye-tracking features. This extracted information can then serve as input data for subsequent fixation estimation and analysis processes in a hybrid wireless eye-tracking system.

[0162] In one embodiment, the hybrid wireless eye-tracking system based on multi-sensor fusion further includes a gaze estimation module configured to determine the gaze direction based on data directly from the data fusion module or the feature extraction module, in conjunction with input from the motion blur compensation unit. The gaze estimation module can receive processed data from multiple sources to perform accurate gaze tracking.

[0163] In a preferred embodiment, the gaze estimation module includes a convolutional neural network (CNN) and a gated recurrent unit (GRU) for gaze trajectory prediction. The CNN component can analyze spatial features extracted from sensor data, while the GRU can process time series data to predict gaze trajectories that change over time.

[0164] The gaze estimation module can combine inputs from the feature extraction module, which may include information about pupil size, gaze duration, and microsaccade frequency. Additionally, this module may incorporate motion-compensated data from a high-speed panoramic camera to provide environmental context for gaze estimation.

[0165] In a more preferred embodiment, the CNN within the gaze estimation module applies multiple convolutional layers to extract relevant features from data from the data preprocessing module or feature extraction module, as well as the input data from the motion blur compensation unit. These convolutional layers can detect patterns and features in the spatial domain that indicate the gaze direction.

[0166] In a further embodiment, the GRU component of the gaze estimation module can process feature sequences over time. In practice, the GRU maintains an internal state that allows it to capture temporal dependencies in eye movement patterns. This cyclical structure may enable the module to predict future gaze trajectories based on past and current inputs.

[0167] The gaze estimation module can fuse the outputs from CNN and GRU components to produce a final estimate of the gaze direction. In practice, the fusion process involves a weighted combination of spatial and temporal features to produce a comprehensive gaze prediction.

[0168] The gaze estimation module can also incorporate adaptive mechanisms to improve its gaze estimation accuracy over time. In one embodiment, a hybrid wireless eye-tracking system based on multi-sensor fusion can use feedback from a high-speed panoramic camera or other sensors to calibrate and improve the gaze estimation model during use.

[0169] By leveraging the combined capabilities of spatial analysis using CNNs and temporal processing using GRUs, the gaze estimation module can provide accurate and robust gaze direction prediction. This gaze estimation method enables hybrid wireless eye-tracking systems to handle complex eye movement patterns and adapt to various user behaviors and environmental conditions.

[0170] In a further embodiment, the aforementioned hybrid wireless eye-tracking system based on multi-sensor fusion also includes a wireless transmission module for transmitting gaze tracking data. As shown in Figure 7, the wireless transmission module 802.11ac can be configured to use multiple wireless protocols to transmit data.

[0171] In one embodiment, the wireless transmission module can utilize 5G millimeter wave (mmWave) technology for data transmission. 5G mmWave technology operates at high frequencies (between 30 GHz and 300 GHz). Compared to low-frequency wireless technologies, this high-frequency operation allows for increased bandwidth and reduced latency.

[0172] In another preferred embodiment, the wireless transmission module also includes the IEEE 802.11ac protocol for data transmission. In some implementations, the 802.11ac protocol can operate in the 5 GHz band and provide high-throughput wireless communication.

[0173] By combining 5G mmWave and IEEE 802.11ac technologies, the wireless transmission module offers flexibility in data transmission. In practice, a hybrid wireless eye-tracking system based on multi-sensor fusion can dynamically switch between these transmission methods based on factors such as signal strength, bandwidth requirements, and power consumption. Dual transmission capabilities offer advantages in various use cases. For example, 5G mmWave technology may be suitable for applications requiring extremely low latency and high bandwidth, such as real-time virtual reality. Conversely, IEEE 802.11ac may be more suitable for indoor environments where mmWave signals may have difficulty penetrating walls.

[0174] In a more preferred embodiment, the wireless transmission module can be configured for low-power operation. In this case, the wireless transmission module utilizes techniques such as adaptive power control, adjusting its transmission power based on signal quality and distance to the receiver. The wireless transmission module can also implement energy-saving protocols to reduce energy consumption during periods of inactivity.

[0175] The wireless transmission module can support real-time data transmission of gaze tracking information. In some cases, time-sensitive data packets are prioritized, and a Quality of Service (QoS) mechanism is implemented to ensure consistent, low-latency transmission of critical eye-tracking data.

[0176] By leveraging the advantages of 5G mmWave and IEEE 802.11ac technologies, the wireless transmission module can provide robust, flexible, and efficient data transmission capabilities for hybrid wireless eye-tracking systems. This dual-mode approach enables the system to adapt to various environmental conditions and application requirements while maintaining high performance and energy efficiency.

[0177] In one embodiment, the aforementioned hybrid wireless eye-tracking system based on multi-sensor fusion further includes an application layer with integrated edge computing capabilities to enable various real-time eye-tracking applications. As shown in Figures 6 and 8, the application layer can receive eye-tracking data transmitted via a wireless transmission module.

[0178] In some cases, the application layer can implement dynamic user interface adaptation based on real-time gaze behavior. The system can analyze incoming gaze tracking data to determine where the user is looking on the display or in the environment. Based on this analysis, the application layer can dynamically adjust user interface elements to optimize the user experience. For example, the application layer can zoom in or highlight the interface element the user is currently viewing, or move important information to areas where the user's gaze is frequently directed. In some implementations, the system can use predictive algorithms to predict where the user is likely to look next and preload content in those areas for faster response times.

[0179] The wireless transmission capabilities of the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in the above embodiments can facilitate various applications in artificial intelligence (AI), ophthalmology, and human-computer interaction (HCI). In some cases, the low-latency transmission of gaze tracking data may enable real-time AI analysis of eye movements for medical diagnosis or user behavior research.

[0180] For ophthalmic applications, this system can provide continuous monitoring of eye movements and pupillary responses, potentially allowing for early detection of certain eye conditions or assessment of treatment effectiveness. In HCI environments, real-time gaze data may enable new forms of hands-free computer control or augmented reality interaction.

[0181] Application layers can leverage edge computing resources to process gaze-tracking data locally, reducing latency and enhancing privacy. In some implementations, edge AI models can run on local devices, further processing eye movements before transmitting the initial analysis of aggregated results to cloud services. This edge computing approach may allow for more responsive applications, as certain gaze-based interactions can be processed and manipulated locally without requiring round-trip communication with remote servers. Furthermore, processing sensitive eye-tracking data at the edge can help address privacy concerns by reducing the amount of raw data transmitted over the network.

[0182] Integrating wireless transmission modules with AI edge cloud applications enables hybrid processing models. In some cases, the system might perform initial gaze analysis on the edge device and then leverage cloud-based AI for more complex or resource-intensive computations. This hybrid approach allows for scalable and flexible gaze-driven applications that can adapt to varying computational needs and network conditions.

[0183] By combining real-time gaze tracking capabilities with edge computing and flexible wireless transmission, hybrid wireless eye-tracking systems can support a wide range of applications across multiple fields. The system can provide low-latency, context-aware gaze data, potentially enabling human-computer interaction and gaze-based analytics in areas such as healthcare, user experience design, and augmented reality.

[0184] The working principle of this system will be explained in detail below through a brief description of the specific execution steps and processes of the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in this embodiment:

[0185] After the data acquisition module acquires data, the data preprocessing module constructs a graphical representation of the Q-Var sensor signal. This graph construction step may include treating each time sample as a node and establishing connections between nodes based on signal similarity or temporal adjacency. After graph construction, a graph Fourier transform (GFT) can be applied to the Q-Var sensor data to convert the time-domain signal into the graph frequency domain. In some cases, this transformation may allow the separation of low-frequency gaze from high-frequency noise and microsaccades. Furthermore, the GFT process involves calculating the Laplacian quantity of the graph, which helps define the signal smoothness on the graph. In some implementations, the system can simultaneously calculate the standard and normalized forms of the graph's Laplacian determinant. Following the GFT, for IMU data, a Kalman filter is used to perform Kalman filtering on the IMU data to eliminate high-frequency noise components. This filtering process may involve applying a Kalman filter in the graph frequency domain to maintain smooth gaze while eliminating unwanted high-frequency variations. Additionally, adaptive thresholding spectral clustering can be performed, a clustering method that can dynamically classify different eye movement events, such as gaze, saccades, and blinks, based on the frequency components of the graph.

[0186] By implementing this graphical Fourier transform-based method and other data preprocessing techniques, a hybrid wireless eye-tracking system based on multi-sensor fusion can effectively process Q-Var sensor data, IMU measurement data, and high-speed panoramic camera data to extract meaningful eye-movement information. This processed data may contribute to accurate gaze estimation and enable various applications in fields such as human-computer interaction and medical diagnosis.

[0187] After the data preprocessing module performs the data preprocessing actions, the preprocessed data from the Q-Var sensor, inertial measurement unit (IMU), and high-speed panoramic camera are combined for comprehensive gaze estimation.

[0188] The hybrid wireless eye-tracking system based on multi-sensor fusion integrates multiple components to provide accurate, real-time gaze tracking. The system combines data from Q-Var sensors, an inertial measurement unit (IMU), and a high-speed panoramic camera to generate comprehensive eye-tracking information.

[0189] In some cases, the system can implement a graph-based approach to combine preprocessed data from Q-Var sensors, IMUs, and high-speed panoramic cameras. This sensor data fusion process may involve constructing a graph representation of the multimodal data, where nodes represent different sensor inputs and edges represent relationships between these inputs. Graph-based fusion methods can more effectively integrate heterogeneous sensor data, potentially improving the overall accuracy of gaze estimation.

[0190] The data fusion module can apply graph-based algorithms to analyze interconnected sensor data. In some implementations, this may involve using graph spectral analysis techniques to identify key features and patterns between different sensor modes. Graph-based methods enable the system to capture complex dependencies between eye movements, head position, and environmental background.

[0191] After sensor data fusion, the gaze estimation module can process the merged data to determine the user's gaze direction. The estimated gaze information can then be wirelessly transmitted using the system's dual-mode transmission capability, as shown in Figures 6 and 8.

[0192] A hybrid wireless eye-tracking system based on multi-sensor fusion can combine its dual-mode transmission module with a power-saving solution. As shown in Figures 6 and 8, the system can simultaneously utilize 5G millimeter-wave technology and the IEEE 802.11ac protocol for wireless data transmission.

[0193] In some cases, energy-saving solutions may involve dynamically switching between two transmission modes based on various factors. The system can assess current power levels, data transmission requirements, and environmental conditions to determine the most energy-efficient transmission method to use at any given time.

[0194] 5G millimeter wave technology may offer high bandwidth and low latency, which could be advantageous for rapidly transmitting large amounts of eye-tracking data. However, this technology may consume more power compared to other wireless protocols. In some implementations, systems can reserve 5G millimeter wave transmissions that require high data throughput or extremely low latency. Conversely, the IEEE 802.11ac protocol may offer a more energy-efficient option for data transmission in certain situations. In some cases, when lower bandwidth is sufficient or when saving battery life is a priority, the system may switch to 802.11ac.

[0195] This power-saving solution can also include an adaptive power control mechanism. In some implementations, the system can adjust the transmission power based on the distance between the eye-tracking device and the receiving unit. When the receiver is nearby, by reducing the transmission power, the system can save energy without compromising data integrity. In some cases, the system may enter a sleep mode or a low-power state during inactivity. The eye-tracking sensor and wireless transmission module can enter these power-saving states when not actively collecting or transmitting data, thereby further extending battery life.

[0196] Dual-mode transmission capability can improve energy efficiency by allowing the system to optimize its power consumption according to the specific needs of different eye-tracking applications. For example, applications requiring continuous high-frequency data transmission may utilize 5G millimeter-wave technology, while less demanding tasks may rely on the more energy-efficient 802.11ac protocol.

[0197] In some implementations, the system can employ intelligent scheduling algorithms to balance power consumption and performance. These algorithms analyze usage patterns and application requirements to predict the optimal time to switch between transmission modes or enter a low-power state.

[0198] This power-saving solution can also leverage edge computing capabilities to reduce the amount of data that needs to be transmitted wirelessly. In some cases, preliminary processing of eye-tracking data can be performed locally, with only the relevant results or compressed data transmitted, thereby reducing the overall power consumption associated with wireless communication.

[0199] By combining these energy-saving technologies with the flexibility of dual-mode transmission, hybrid wireless eye-tracking systems can achieve a balance between energy efficiency and high performance. This approach extends the runtime of battery-powered devices while maintaining the ability to provide high-quality eye-tracking data when needed.

[0200] The system's application layer can receive transmitted gaze tracking data and utilize edge computing resources from various applications. In some cases, the application layer can use evolutionary game theory to implement adaptive, personalized eye comfort modes. These eye comfort modes can dynamically adjust display parameters or content presentation based on the user's gaze pattern and environmental conditions.

[0201] Evolutionary game-theoretic approaches to eye comfort modes may involve treating different comfort-related parameters as strategies in a game. The system can iteratively update these strategies based on user feedback and gaze behavior, evolving towards the optimal comfort configuration over time. This adaptive mechanism could allow the system to provide a personalized eye-tracking experience for individual users, while taking into account constantly changing environmental factors.

[0202] In summary, the overall data flow of a hybrid wireless eye-tracking system based on multi-sensor fusion can be summarized as follows: preprocessed sensor acquisition, sensor fusion, gaze estimation, wireless transmission, and finally application-level processing. Each stage in this pipeline contributes to the system providing accurate, real-time eye-tracking data.

[0203] In some implementations, the system can use feedback loops between different components to continuously improve and enhance performance. For example, the application layer can provide feedback to the sensor fusion module to adjust fusion parameters based on application-specific needs or detected gaze patterns.

[0204] Graph theory-based sensor fusion methods, combined with evolutionary game theory-driven eye comfort patterns, can enable hybrid wireless eye-tracking systems to adapt to different usage scenarios and user needs. These advanced processing technologies, along with the system's wireless capabilities and edge computing integration, may allow for flexible and efficient eye-tracking applications across various domains.

[0205] It is understood that the hybrid wireless eye-tracking system based on multi-sensor fusion proposed in this embodiment of the invention corresponds to the hybrid wireless eye-tracking method based on multi-sensor fusion described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the hybrid wireless eye-tracking method based on multi-sensor fusion, and will not be repeated here.

[0206] Example 3:

[0207] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any of the above embodiments and their preferred embodiments. The method mainly includes:

[0208] S1. Collect comprehensive data on user eye movement and head position based on multiple sensors including Q-Var sensor, inertial measurement unit and high-speed panoramic camera;

[0209] S2. Perform data fusion on the comprehensive data;

[0210] S3. Estimate the user's gaze direction based on the fused sensor data.

[0211] It is understood that the hybrid wireless eye-tracking electronic device based on multi-sensor fusion provided in the embodiments of the present invention corresponds to the hybrid wireless eye-tracking method and system based on multi-sensor fusion described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the hybrid wireless eye-tracking method and system based on multi-sensor fusion, and will not be repeated here.

[0212] Example 4:

[0213] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any of the foregoing embodiments and their preferred embodiments, the method comprising:

[0214] S1. Collect comprehensive data on user eye movement and head position based on multiple sensors including Q-Var sensor, inertial measurement unit and high-speed panoramic camera;

[0215] S2. Perform data fusion on the comprehensive data;

[0216] S3. Estimate the user's gaze direction based on the fused sensor data.

[0217] It is understood that the hybrid wireless eye-tracking storage medium based on multi-sensor fusion provided in the embodiments of the present invention corresponds to the hybrid wireless eye-tracking method and system based on multi-sensor fusion described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the hybrid wireless eye-tracking method and system based on multi-sensor fusion, and will not be repeated here.

[0218] In summary, compared with existing technologies, it has the following beneficial effects:

[0219] 1. This application proposes a hybrid wireless eye-tracking technology based on multi-sensor fusion. First, it collects comprehensive data on the user's eye movements and head position using multiple sensors, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera. Then, it fuses this comprehensive data. Finally, it estimates the user's gaze direction based on the fused sensor data. This method improves eye-tracking performance under different environmental conditions, achieving higher accuracy and better stability.

[0220] 2. This application proposes a hybrid wireless eye-tracking technology based on multi-sensor fusion, which performs graphical Fourier transform on the data acquired by the Q-Var sensor; and / or Kalman filtering on the data acquired by the inertial measurement unit; and / or motion blur compensation on the data acquired by the high-speed panoramic camera; and applies spectral clustering technology to further process the Q-Var data after graphical Fourier transform. By processing sensor-specific noise and artifacts in the data preprocessing stage, this application contributes to the overall accuracy and robustness of the hybrid wireless eye-tracking system proposed in this application.

[0221] 3. The hybrid wireless eye-tracking technology based on multi-sensor fusion proposed in this application utilizes a graph theory-based method to combine data from Q-Var sensors, IMUs, and high-speed panoramic cameras, which can more effectively integrate heterogeneous sensor data and potentially improve the overall accuracy of gaze estimation.

[0222] 4. The hybrid wireless eye-tracking technology based on multi-sensor fusion proposed in this application can provide rich eye movement features by extracting these key features—pupil size, fixation time, and microsaccade frequency—from preprocessed sensor data.

[0223] 5. This application proposes a hybrid wireless eye-tracking technology based on multi-sensor fusion. By leveraging the combined capabilities of spatial analysis using CNNs and temporal processing using GRUs, gaze estimation can provide accurate and robust gaze direction prediction. This gaze estimation method enables the hybrid wireless eye-tracking system to handle complex eye movement patterns and adapt to various user behaviors and environmental conditions.

[0224] 6. This application proposes a hybrid wireless eye-tracking technology based on multi-sensor fusion. By leveraging the advantages of 5G mmWave and IEEE 802.11ac technologies, the wireless transmission method can provide robust, flexible, and efficient data transmission capabilities for the hybrid wireless eye-tracking system. This dual-mode approach enables the system to adapt to various environmental conditions and application requirements while maintaining high performance and energy efficiency.

[0225] 7. This application proposes a hybrid wireless eye-tracking technology based on multi-sensor fusion. This technology enables dynamic user interface adaptation based on estimated user gaze direction and achieves an adaptive personalized eye comfort mode based on game theory. These strategies can be iteratively updated based on user feedback and gaze behavior, evolving towards the optimal comfort configuration over time. This adaptive mechanism may allow the system to provide a personalized eye-tracking experience for individual users, while taking into account constantly changing environmental factors.

[0226] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0227] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid wireless eye-tracking method based on multi-sensor fusion, characterized in that, The method includes: It uses a multi-sensor system, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera, to collect comprehensive data on the user's eye movement and head position. Perform data fusion on the comprehensive data; The user's gaze direction is estimated based on the fused sensor data.

2. The hybrid wireless eye-tracking method based on multi-sensor fusion as described in claim 1, characterized in that, Before performing data fusion on the comprehensive data, the comprehensive data is preprocessed; the preprocessing includes: Perform graphical Fourier transform on the data acquired by the Q-Var sensor; and / or Perform Kalman filtering on the data acquired by the inertial measurement unit; and / or Motion blur compensation is applied to the data collected by the high-speed panoramic camera.

3. The hybrid wireless eye-tracking method based on multi-sensor fusion as described in claim 2, characterized in that, The preprocessing also includes: Adaptive thresholding is performed on Q-Var sensor data after graphical Fourier transform based on spectral clustering technology.

4. The hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of claims 2-3, characterized in that, Data fusion of the aforementioned comprehensive data includes: A graph theory-based method is used to fuse preprocessed data from Q-Var sensors, inertial measurement units, and high-speed panoramic cameras.

5. The hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of claims 1-4, characterized in that, The estimation of the user's gaze direction based on the fused sensor data includes: User gaze trajectory prediction is performed based on the analysis and fusion of sensor data using convolutional neural networks and gated recurrent units.

6. The hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of claims 1-5, characterized in that, The method further includes: Use at least one of 5G millimeter wave technology or the IEEE 802.11ac protocol to wirelessly transmit gaze tracking data.

7. The hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of claims 1-6, characterized in that, The method further includes: Dynamic user interface adaptation is achieved based on the estimated user's gaze direction; and A game theory-based approach to achieve an adaptive, personalized eye comfort mode.

8. A hybrid wireless eye-tracking system based on multi-sensor fusion, characterized in that, The system includes: The data acquisition module is configured to acquire comprehensive data on the user's eye movements and head position based on multiple sensors, including a Q-Var sensor, an inertial measurement unit, and a high-speed panoramic camera. The data fusion module is configured to perform data fusion on the comprehensive data; The gaze estimation module is configured to estimate the user's gaze direction based on the fused sensor data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the hybrid wireless eye-tracking method based on multi-sensor fusion as described in any one of claims 1-7.

Images