Event-Based IR Camera for Low-Latency AR Eye Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented and mixed reality systems face challenges in acquiring information about physical objects with low latency and low power consumption, leading to limitations in size, power consumption, and realism, which affects user enjoyment and utility.
Innovation Solution
The use of image sensors with dynamic vision sensing techniques and patch tracking, which output differential image information only for changes detected in specific regions of the image array, reducing processing requirements and latency, and incorporating plenoptic camera techniques for passive depth measurement without active light sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional image sensors are used to capture full image frames at high rates, then motion detection precision is improved, but power consumption increases and latency increases
Solution Approach 1:
The patent extracts only the essential information needed for motion detection by using event-based sensors that output differential image information only for changes detected in specific regions, rather than capturing complete image frames. This extraction principle reduces data volume and processing requirements, directly lowering power consumption while maintaining motion detection precision.
Solution Approach 2:
The patent segments the image array into multiple regions and applies patch tracking to selectively monitor specific areas of interest. By dividing the field of view into patches and only processing events within those patches, the system reduces the overall processing load and power consumption while maintaining precise motion detection in the regions that matter most.
2Measurement precision
If conventional image sensors capture full image frames at high rates, then motion detection precision is improved, but latency increases
Solution Approach 1:
The patent extracts only the essential information needed for motion detection by using event-based sensors that output differential image information only for changes detected in specific regions, rather than capturing complete image frames. This extraction principle reduces data volume and processing requirements, directly lowering power consumption while maintaining motion detection precision.
Solution Approach 2:
The patent implements event-driven periodic action where image information is captured and processed only when changes are detected, rather than continuously capturing full frames at fixed intervals. This event-based triggering mechanism reduces latency by immediately processing relevant changes while reducing overall processing load.
3Measurement precision
If active light sources are used for depth measurement, then depth measurement precision is improved, but power consumption increases
Solution Approach 1:
The patent applies self-service by using passive plenoptic camera techniques that leverage existing ambient light and optical properties to perform depth measurement without requiring active light sources. The system uses the scene's own light properties and the camera's optical design to extract depth information, eliminating the need for power-consuming active illumination while maintaining depth measurement capability.
4Measurement precision
If high-resolution image arrays are used, then image quality is improved, but processing requirements increase leading to higher latency
Solution Approach 1:
The patent extracts only the essential information needed for motion detection by using event-based sensors that output differential image information only for changes detected in specific regions, rather than capturing complete image frames. This extraction principle reduces data volume and processing requirements, directly lowering power consumption while maintaining motion detection precision.
Solution Approach 2:
The patent segments the image array into multiple regions and applies patch tracking to selectively monitor specific areas of interest. By dividing the field of view into patches and only processing events within those patches, the system reduces the overall processing load and power consumption while maintaining precise motion detection in the regions that matter most.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables AR and MR systems to acquire information with low latency and reduced power consumption, allowing for realistic and efficient rendering of virtual objects in relation to physical objects, improving user experience and system utility.
Implementation Method 1
Each pixel cell of the plurality of pixel cells may comprise a component sensitive to IR light. The at least one event detection circuit may be configured to provide signals indicating a change in sensed IR light
Data Source
AI summary
A high-resolution image sensor suitable for use in an augmented reality (AR) system. The AR system may be small enough to be packaged within a wearable device such as a set of goggles or mounted on a frame resembling ordinary eyeglasses. The image sensor may have pixels configured to output events indicating changes in sensed IR light. Those pixels may be sensitive to IR light of the same frequency source as an active IR light source, and may be part of an eye tracking camera, oriented toward a user's eye. Changes in IR light may be used to determine the location of the user's pupil, which may be used in rendering virtual objects. The events may be generated and processed at a high rate, enabling the system to render the virtual object based on the user's gaze so that the virtual object will appear more realistic to the user.