Retina-Based Eye Tracking for Sub-0.1° Near-Eye Display Gaze Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional eye-tracking methods in head-mounted displays (HMDs) suffer from low accuracy and difficulty in improving tracking resolution beyond 0.5°-1°, with pupil-glint methods being the most widely used but limited in precision, and retinal-based tracking is not easily integrated into HMDs due to complexity and sensor requirements.
Innovation Solution
A real-time localization method for object tracking using retina-based eye tracking, employing a deep learning model with a deep convolutional neural network and a Kalman filter, which generates a mosaicked search image for image registration, enabling accurate gaze estimation and object localization without relying on pupil-glint techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pupil-glint methods are used for eye tracking in HMDs, then the tracking resolution can reach 0.5°-1°, but it is difficult to improve accuracy beyond this limit
Solution Approach 1:
The patent replaces conventional pupil-glint-based optical tracking methods with a retinal imaging approach that captures actual retinal images through the HMD optics. This substitution enables direct observation of retinal features and blood vessels, achieving tracking accuracy better than 0.1° by eliminating the fundamental resolution limits of pupil-glint methods.
Solution Approach 2:
The patent changes the fundamental parameter being tracked from pupil position and glint reflection patterns to retinal image features and blood vessel patterns. By capturing and analyzing retinal imagery directly through the HMD display optics, the system achieves superior tracking precision that overcomes the 0.5°-1° limitation of conventional methods.
2Measurement precision
If retinal-based tracking is integrated into HMDs, then tracking resolution can be improved beyond 0.5°-1°, but the device complexity and sensor requirements increase
Solution Approach 1:
The patent makes the HMD display optics serve a dual function: displaying virtual content to the user and simultaneously capturing retinal images for eye tracking. By utilizing the existing display optics for both purposes, the system achieves high-resolution retinal imaging without adding separate complex sensor systems, thereby improving tracking resolution while minimizing increased device complexity.
Solution Approach 2:
The system uses the HMD's own display optics to capture retinal images, making the display system serve its own eye tracking needs. The display optics naturally focus light onto the retina and reflect it back, allowing the same optical path to be used for both display and retinal imaging, eliminating the need for additional dedicated sensors.
3Ease of manufacture
If conventional pupil-glint methods are used, then the system is easier to implement, but the tracking error remains at 0.5°-1°
Solution Approach 1:
The patent substitutes retinal image capture and analysis for pupil-glint detection. By capturing actual retinal images through the HMD optics and analyzing blood vessel patterns and retinal features, the system achieves tracking error reduced to less than 0.1°, significantly improving precision while maintaining implementation feasibility through software-based image processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for tracking eye movement in an augmented reality system identify a plurality of base images of an object or a portion thereof. A search image may be generated based at least in part upon at least some of the plurality of base images. A deep learning result may be generated at least by performing a deep learning process on a base image using a neural network in a deep learning mode. A captured image may be localized at least by performing an image registration process on the captured image and the search image using a Kalman filter model and the deep learning result.