Depth-Based Scene Camera Retargeting for HMD Viewpoint Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
HMDs often impair distance perception, cause disorientation, and result in poor hand-eye coordination due to the mismatch between the positions of the eyes, display, and camera, leading to distorted real-world images.
Innovation Solution
Implementing a planar world assumption to transform captured images using homography, compensating for the positional differences between the scene camera and user's viewpoint, thereby aligning the image as if captured from the user's perspective.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the scene camera captures images from its fixed position, then the camera can continuously capture the environment, but the displayed image does not match the user's actual viewpoint, causing distance perception impairment and disorientation
Solution Approach 1:
The patent introduces an intermediary computational layer that transforms camera images to match the user's viewpoint. This intermediary processing layer mediates between the fixed camera position and the moving user position, resolving the mismatch without requiring physical repositioning of the camera or display.
Solution Approach 2:
The system dynamically changes image parameters (position, orientation, scale) based on the user's head position and orientation relative to the camera. By adjusting these parameters in real-time, the system maintains accurate distance perception despite the fixed camera position.
2Ease of operation
If the display shows the raw camera image, then the system is simple and fast, but hand-eye coordination is poor due to the mismatch between camera view and user view
Solution Approach 1:
The system performs preliminary actions by pre-calculating transformation parameters based on predicted user movement patterns and preparing transformed images in advance. This reduces the real-time processing burden while maintaining accurate hand-eye coordination.
Solution Approach 2:
The patent replaces mechanical solutions (such as moving the camera or display physically) with computational image transformation. This substitution achieves the same effect of aligning views without the mechanical complexity and time delays associated with physical movement.
3Measurement precision
If multiple cameras are used to capture the environment from different positions, then the viewpoint mismatch can be reduced, but the device complexity and cost increase significantly
Solution Approach 1:
Instead of using multiple physical cameras, the system creates a computational copy of the scene by transforming the single camera's image. This digital copy is then adjusted to match the user's viewpoint, achieving multi-viewpoint functionality without the hardware complexity of multiple cameras.
Data Source
AI summary
In one implementation, a method of scene camera retargeting is performed by a head-mounted device (HMD) including one or more processors, non-transitory memory, a scene camera, and a display. The method includes capturing, using the scene camera, an image of a scene. The method includes determining, using the one or more processors, a depth of the scene. The method further includes transforming, using the one or more processors, the image of the scene based on the depth and a difference between a point-of-view of the scene camera and a point-of-view of a user. The method further includes displaying, on the display, the transformed image.