MR Pass-Through View Texture Reconstruction for Eye-Camera Misalignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The issue of missing texture in mixed reality (MR) pass-through views due to the misalignment between the HMD's front camera and the user's eyes, leading to poor visual experience.
Innovation Solution
A method involving a host device that utilizes historical and current per-frame information to determine historical tracking and texture information of a target object, predicts current tracking information, renders an object image based on this information, and generates a pass-through view by superimposing the rendered image onto the current frame, thereby addressing the missing texture problem.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the front camera is positioned at the user's eye location, then the pass-through view can capture all visible textures, but the HMD structure cannot accommodate the camera at the correct position
Solution Approach 1:
The patent introduces an intermediary computational process that bridges the gap between the fixed camera position and the user's eye position. By using multi-view geometry and ray-tracing algorithms, the system virtually reconstructs the scene as if captured from the eye position, eliminating the need for physical camera relocation while achieving accurate texture capture.
Solution Approach 2:
The patent creates a virtual copy of the scene from the camera's perspective and transforms it to match the user's eye position. By generating synthetic image frames that replicate what the eye would see, the system avoids the complexity of positioning the physical camera at the eye location while achieving the same visual fidelity.
2Device complexity
If the front camera is positioned away from the user's eyes, then the HMD structure is simplified, but texture information is lost in the pass-through view
Solution Approach 1:
The patent replaces the mechanical solution of positioning the camera at the eye location with a computational approach. By using image processing algorithms, multi-view geometry, and ray-tracing, the system recovers and synthesizes texture information that would otherwise be lost, eliminating the need for complex mechanical camera positioning while preserving all visual details.
Solution Approach 2:
The patent transitions from a single-camera 2D capture to a multi-view 3D reconstruction approach. By capturing images from multiple angles and using depth information, the system can synthesize views from any position in 3D space, including the user's eye position, thereby recovering texture information without physical camera relocation.
3Measurement precision
If real-time image processing is performed to correct the view, then the texture accuracy is improved, but the processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing multi-view image data and depth information during idle periods or in the background. When real-time processing is needed, the system retrieves and combines these pre-processed data structures, significantly reducing the computational burden and processing time while maintaining high texture accuracy.
Solution Approach 2:
The patent implements continuous background processing that constantly updates the multi-view image data and depth maps as the user moves. This ensures that the pre-computed data is always current and ready for immediate use, eliminating the need for heavy real-time computation while maintaining accurate texture representation during active viewing.
Data Source
AI summary
The embodiments of the disclosure provide a method for improving a pass-through view and a host. The method includes: obtaining current per-frame information and historical per-frame information of the host; determining historical tracking information and texture information of a target object based on the historical per-frame information of the host; predicting current tracking information of the target object based on the historical tracking information of the target object and the current per-frame information of the host; rendering an object image corresponding to the target object based on the current tracking information and the texture information of the target object; and generating the pass-through view based on the rendered object image, the current per-frame information of the host, and the current tracking information of the target object, and displaying the pass-through view.