XR Skybox Generation From 2D Images Using GAN Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to support non-tech-savvy users in creating customizable skyboxes for their XR worlds, leading to a lack of familiarity and engagement in XR environments.
Innovation Solution
A method involving splitting a user-selected 2D image into portions, mapping these portions onto a 3D shape, and using a generative adversarial network to interpolate and fill in unmapped areas, creating a skybox for the XR world.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 360-degree video is used to create immersive virtual reality environments, then immersion quality is improved, but device complexity and data processing requirements increase significantly
Solution Approach 1:
The patent uses pre-rendered 360-degree panoramic images as static copies of the virtual environment from multiple viewpoints. Instead of processing real-time 360-degree video data, the system uses these copied images to simulate the immersive experience, significantly reducing computational complexity while maintaining visual quality.
Solution Approach 2:
The patent segments the 360-degree panoramic image into multiple viewports (front, back, left, right hemispheres) that can be independently processed and mapped to different faces of a cube or sphere. This segmentation allows the system to handle complex 360-degree content through simpler, manageable sections.
2Ease of operation
If 360-degree video data is processed in real-time, then interactivity is improved, but processing time and computational resources increase
Solution Approach 1:
The patent performs all heavy processing in advance by pre-rendering 360-degree panoramic images from multiple camera positions and organizing them into a structured format. This preliminary action eliminates the need for real-time processing during user interaction, allowing instant retrieval and display of appropriate views based on user orientation.
Solution Approach 2:
The patent replaces the mechanical/video processing system with an image processing system. Instead of processing continuous video frames, the system uses static panoramic images that can be quickly manipulated and displayed, substituting complex temporal processing with simpler spatial transformations.
3Loss of information
If multiple camera positions are used to capture full 360-degree views, then coverage completeness is improved, but data volume and processing complexity increase
Solution Approach 1:
The patent merges images from multiple camera positions into a single 360-degree panoramic image. By combining the fields of view from multiple cameras into one comprehensive panorama, the system achieves complete coverage without storing separate datasets for each camera position, thus reducing overall data volume.
Solution Approach 2:
The patent transitions from three-dimensional video data to two-dimensional panoramic images. This dimensional reduction allows complete 360-degree coverage to be achieved in a planar format that requires less storage and processing resources compared to volumetric video data.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Aspects of the present disclosure are directed to creating a skybox for an artificial reality ("XR") world from a two-dimensional ("2D") image. The 2D image is scanned and split into at least two portions. The portions are mapped onto the interior of a virtual enclosed 3D shape, for example, a virtual cube. A generative adversarial network (GAN) interpolates from the information in the areas mapped from the portions to fill in at least some unmapped areas of the interior of the 3D shape. The 3D shape can be placed in a user's XR world to become the skybox surrounding that world.