360-Degree Mixed Reality Environment Generation via Video Stitching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mixed reality technologies struggle to provide a seamless and immersive 360-degree view of real-world environments, limiting user interaction and realism due to incomplete or disjointed virtual and real-world object interactions.
Innovation Solution
The system generates a 360-degree mixed reality environment by combining real-world videos from multiple cameras, creating a 3D environment with occlusion and background layers, allowing virtual objects to interact with real-world objects based on 3D space locations, and adapting to user movements by stitching frames and storing depth and motion information for optimized rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple real-world videos from different cameras are combined to create a 360-degree view, then the completeness and immersion of the virtual environment is improved, but the system complexity and computational requirements increase
Solution Approach 1:
The system divides the 360-degree environment into multiple video segments captured by different cameras, each covering a specific field of view. These segmented videos are then stitched together to form the complete panoramic environment, allowing manageable processing of each segment while achieving comprehensive coverage.
Solution Approach 2:
The system transitions from 2D video frames to 3D spatial representation by generating depth maps and 3D point clouds from multiple video angles. This dimensional transformation enables virtual objects to be positioned and interacted with in three-dimensional space within the panoramic environment.
2Measurement precision
If depth information and motion vectors are stored for every pixel in video frames, then the precision of 3D object interaction is improved, but the data storage requirements and processing load increase
Solution Approach 1:
The system calculates depth information and motion vectors for all pixels during processing, but only stores these data for a selected subset of pixels that are sufficient to represent the 3D structure and enable accurate virtual object interactions. This partial storage approach maintains precision while reducing data volume.
3Reliability
If transparent 3D objects are generated to replicate real-world objects for occlusion effects, then the realism of virtual-real object interaction is improved, but the rendering complexity and computational resources increase
Solution Approach 1:
The system creates transparent 3D copies of real-world objects by generating depth maps and 3D models from video frames. These copied objects are rendered with transparency to simulate occlusion effects, allowing virtual objects to realistically interact with and be occluded by replicated real-world elements without requiring full photorealistic reconstruction.
Data Source
AI summary
Systems and methods for generating a 360 degree mixed virtual reality environment that provides a 360 degree view of an environment in accordance with embodiments of the invention are described. In a number of embodiments, the 360 degree mixed virtual reality environment is obtained by (1) combining one or more real world videos that capture images of an environment with (2) a virtual world environment that includes various synthetic objects that may be placed within the real world clips. Furthermore, the virtual objects embedded within the 360 degree mixed reality environment interact with the real world objects depicted in the real world environment to provide a realistic mixed reality experience.