Single-View 2.5D XR Imaging Using Depth Map Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing extended reality (XR) systems primarily display two-dimensional images, failing to leverage available 3D spaces and providing a suboptimal immersive experience.
Innovation Solution
A method and system that converts single-view 2D images into 2.5D views using machine learning models to generate texture and depth maps, enabling stereoscopic image pairs with depth information for XR applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 2D images are displayed in existing XR systems, then device complexity is reduced and processing is simplified, but immersive experience and depth perception deteriorate
Solution Approach 1:
The patent applies dimensionality change by transforming 2D images into 2.5D representations through the generation of depth maps and multiple viewpoint images. The machine learning model processes a single 2D input image to produce depth information and synthesize additional viewpoint images, effectively adding a depth dimension without requiring full 3D scanning equipment. This resolves the contradiction by providing immersive 2.5D experience while keeping the processing pipeline based on simple 2D image input.
2Loss of information
If multiple viewpoint images are generated using traditional multi-camera systems, then depth information is improved, but device complexity and cost increase
Solution Approach 1:
The patent uses copying by generating synthetic viewpoint images through machine learning rather than capturing them with physical cameras. The model learns to copy and transform the single input image into multiple viewpoint representations with depth information, eliminating the need for multiple physical cameras or complex multi-camera systems. This approach preserves depth information while significantly reducing system complexity.
Solution Approach 2:
The patent replaces the mechanical multi-camera system with a computational machine learning model. Instead of using multiple physical imaging sensors to capture depth and multiple viewpoints simultaneously, the system uses a trained neural network to synthesize this information from a single 2D image, substituting mechanical complexity with intelligent algorithms.
3Productivity
If 2D images are used in XR applications, then processing speed is maintained, but user immersion and engagement deteriorate
Solution Approach 1:
The patent applies preliminary action by pre-training the machine learning model on large datasets of images and depth maps. Once trained, the model can rapidly generate 2.5D content from new 2D images in real-time or near-real-time. The computationally intensive training phase is performed beforehand, allowing fast inference during actual XR application usage. This maintains productivity during content generation while providing immersive 2.5D experience to users.
Data Source
AI summary
A method includes obtaining a 2D image captured using an imaging sensor. The 2D image is associated with an imaging sensor pose. The method also includes providing the 2D image, the imaging sensor pose, and one or more additional imaging sensor poses to at least one machine learning model that is trained to generate a texture map and a depth map for the imaging sensor pose and each additional imaging sensor pose. The method further includes generating a stereo image pair based on the texture maps and the depth maps. The stereo image pair represents a 2.5D view of the 2D image. The 2.5D view includes a pair of images each including multiple collections of pixels and, for each collection of pixels, a common depth associated with the pixels in the collection of pixels. In addition, the method includes initiating display of the stereo image pair on an XR device.