Depth Map Refinement for XR Perspective Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In extended reality (XR) environments, head-mounted devices (HMDs) often present users with images of the physical environment that do not accurately reflect what they would see without the device, due to positional differences between the eyes, display, and camera, leading to user discomfort, poor distance perception, and disorientation.
Innovation Solution
The method involves capturing images of the physical environment with a scene camera, generating depth maps to align pixel depths with the user's perspective, and transforming the images to appear as if captured from the user's eye location, using static or dynamic depth maps to reduce artifacts and improve spatial and temporal accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the camera position is fixed in the HMD, then the device structure is simplified, but the presented image does not match the user's natural viewpoint causing discomfort and disorientation
Solution Approach 1:
The patent replaces the mechanical approach of physically positioning the camera at the user's eye location with a computational approach. The system captures images from the fixed camera position and uses image transformation algorithms to synthetically generate what the view would look like from the user's eye position, thereby resolving the contradiction between simplified device structure and user comfort.
2Ease of operation
If the camera is positioned away from the user's eye, then the HMD design is more practical and comfortable to wear, but the captured image perspective differs from what the user would naturally see
Solution Approach 1:
The patent introduces an intermediary computational process between the camera capture and the final display. The image transformation algorithm acts as a mediator that translates the image from the camera's perspective to the user's eye perspective, allowing the camera to be positioned comfortably away from the eye while maintaining perspective accuracy.
3Speed
If a simple depth map is used for image transformation, then the processing speed is faster, but spatial artifacts and distortion occur in the transformed image
Solution Approach 1:
The patent employs dynamic depth map refinement where the depth map is continuously updated and refined based on the specific image being transformed. Rather than using a static, simple depth map, the system adapts the depth information to match the current scene geometry, maintaining high transformation accuracy while managing processing throughput through selective refinement.
4Manufacturing precision
If the depth map is refined to reduce spatial artifacts, then the image transformation accuracy improves, but the processing time and computational load increase
Solution Approach 1:
The patent applies local quality refinement to the depth map, focusing computational resources on specific regions of the image where depth accuracy is most critical for reducing artifacts. Rather than uniformly refining the entire depth map, the system identifies and refines only the necessary local regions, thereby improving transformation accuracy while minimizing additional processing time.
Data Source
AI summary
In one implementation, a method of performing perspective correction is performed by a device including an image sensor, a display, one or more processors, and a non-transitory memory. The method includes: capturing, using the image sensor, an image of a physical environment; obtaining a first depth map including a plurality of depths respectively associated with a plurality of pixels of the image of the physical environment; generating a second depth map by aligning one or more portions of the first depth map based on a control signal associated with the image of the physical environment; transforming, using the one or more processors, the image of the physical environment based on the second depth map; and displaying, via the display, the transformed image.