Depth Image Fusion Using PDAF Pixels for Gaze Region Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional depth cameras in XR headsets suffer from low resolution, especially near depth discontinuities, leading to poor quality images and misalignment issues, and existing PDAF pixels are not suitable for precise depth measurement on flat surfaces, hindering accurate depth information capture.
Innovation Solution
A method and apparatus that combines processed PDAF pixels from a color camera sensor with depth maps from one or more depth sensors to enhance depth information, utilizing the complementary strengths of both types of sensors without affecting frame rate, and employing triangulation to correct misalignments and denoise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional depth cameras (ToF/LIDAR) are used to capture depth information, then depth sensing is achieved, but resolution is low especially near depth discontinuities leading to poor image quality
Solution Approach 1:
The patent combines data from multiple depth sensing technologies (ToF camera and LiDAR) with color camera PDAF pixels to create a fused depth map. This merging compensates for individual sensor limitations, particularly improving depth information quality at edges and discontinuities where single sensors fail.
Solution Approach 2:
The patent creates a composite depth mapping approach by integrating multiple depth sensing modalities (active infrared ToF, optical LiDAR, and passive PDAF) into a unified depth representation. This composite approach leverages the complementary strengths of each sensing type to overcome individual weaknesses.
2Reliability
If depth cameras are positioned at different physical locations from color cameras to enable depth sensing, then depth capture capability is achieved, but misalignment occurs between depth maps and color imagery
Solution Approach 1:
The patent introduces an intermediary reprojection process that maps depth information from the depth camera coordinate system to the color camera coordinate system. This intermediary transformation resolves the misalignment issue caused by physical separation of sensors while preserving depth sensing reliability.
Solution Approach 2:
The patent transforms depth coordinates and applies geometric transformations to reconcile the different physical positions of depth and color cameras. By changing the coordinate parameters and applying alignment transformations, the system achieves precise registration despite physical separation.
3Quantity of substance
If PDAF pixels are used for autofocus in mobile phone cameras, then absolute distance information is provided, but they lack the ability to precisely compute exact distance to objects
Solution Approach 1:
The patent merges PDAF pixel data with depth map information from dedicated depth sensors to compensate for PDAF's imprecision. The combination allows the system to maintain broad depth coverage while achieving precise distance measurements through fusion with more accurate depth sensing data.
4Manufacturing precision
If reprojection process is employed to align depth map with color camera viewpoint, then misalignment issue is addressed, but the process may fail due to disocclusion
Solution Approach 1:
The patent creates a composite depth mapping approach that integrates multiple depth sensing modalities (active infrared ToF, optical LiDAR, and passive PDAF) into a unified depth representation. This composite approach leverages the complementary strengths of each sensing type to overcome individual weaknesses, including robustness to disocclusion.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Provides more reliable 3D information in the gaze region by combining PDAF pixels with depth maps, improving depth accuracy and autofocus without misalignment, especially on flat surfaces, and enhancing depth image quality.
Implementation Method 1
Phase Detection Autofocus (PDAF) pixels are used for autofocus in mobile phone cameras, measuring the optical distance to a given object
Implementation Method 2
Time-of-Flight (ToF) depth cameras
Implementation Method 3
Light Detection and Ranging (LIDAR) depth cameras
Data Source
AI summary
A method for depth image enhancement implemented in at least one apparatus includes reading and processing Phase Detection Autofocus (PDAF) pixels of a region-of-interest (ROI) area of a gaze region in an image obtained from a color camera sensor; utilizing one or more depth camera sensors to provide one or more depth maps of the ROI area of the gaze region; and combining the processed PDAF pixels of the ROI area of the gaze region and the one or more depth maps of the ROI area of the gaze region to obtain an updated ROI area with complementary depth information of the ROI area of the gaze region.