360 Video Capture for Navigable Virtual Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies for creating photorealistic and navigable virtual environments face challenges in achieving real-time interactivity and cinematic quality due to limitations in data compression and rendering of 360-degree videos, which restrict user navigation and interaction.
Innovation Solution
A method and apparatus for simultaneously capturing and preprocessing 360-degree video and audio data from multiple viewpoints, compressing it into a three-dimensional representation, and rendering a virtual environment that allows seamless navigation and interactivity by combining captured data with real-time rendering, using techniques like multi-stream video compression and spatial audio encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If 360 degree videos are used to capture real environments, then photorealism is improved, but user navigation is restricted to the location of capture
Solution Approach 1:
The system segments the environment into multiple overlapping 360-degree video clips captured from different locations. Each clip represents a localized volumetric capture, and the system stitches these segments together to create a navigable 3D environment where users can move between capture points while maintaining photorealistic quality in each segment.
Solution Approach 2:
The system transitions from 2D equirectangular projection to 3D volumetric representation by extracting depth information and camera parameters from multiple 360-degree videos. This dimensional transformation enables users to navigate freely in three-dimensional space rather than being constrained to a single fixed viewpoint.
2Adaptability or versatility
If computer graphics rendering is used to create virtual environments, then navigability is improved, but photorealism deteriorates due to computational cost
Solution Approach 1:
Instead of generating photorealistic graphics through computational rendering, the system captures actual photorealistic video footage from multiple viewpoints and uses computer vision algorithms to reconstruct the 3D geometry. This copying approach preserves the inherent photorealism of real-world lighting and textures while enabling navigability through virtual camera movement between capture points.
Solution Approach 2:
The system replaces traditional computer graphics rendering pipelines with a computer vision-based reconstruction pipeline. Instead of synthesizing images through ray tracing or rasterization, the system uses photogrammetry and structure-from-motion algorithms to derive 3D models from captured video frames, substituting mechanical rendering processes with optical capture and computational reconstruction.
3Manufacturing precision
If volumetric capture systems use multiple camera arrays to capture a volume from all directions, then photorealism is improved, but data compression becomes a bottleneck limiting the number of assets that can be rendered
Solution Approach 1:
The system performs preliminary compression and preprocessing of volumetric data during the capture phase. By encoding video streams and extracting camera parameters in advance, the system reduces the data volume before storage and transmission, enabling faster loading and rendering of multiple assets without compromising photorealistic quality during playback.
Solution Approach 2:
The system dynamically loads and streams volumetric assets based on user navigation needs rather than pre-loading all assets. As users move between capture points, the system selectively loads only the necessary video clips and depth maps for the current viewpoint, optimizing bandwidth utilization and rendering throughput while maintaining photorealistic quality.
4Ease of manufacture
If mesh-based implementations are used for volumetric capture, then integration into existing rendering software is improved, but a substantial portion of mesh data is not used during rendering from a given viewer's perspective
Solution Approach 1:
The system extracts only the essential elements needed for rendering from the complete volumetric dataset. Instead of transmitting entire mesh structures with all vertices and faces, the system extracts and transmits only the video clips and camera parameters relevant to the current viewpoint, eliminating unnecessary mesh data while maintaining rendering quality and reducing data volume.
Data Source
AI summary
Systems, methods, apparatuses, and computer program products for creating freely explorable, dynamic and photorealistic virtual environments, reconstructing view dependent holograms in real-time, and inserting 3D virtual objects into 360 camera based navigable environment. A method, may include simultaneously capturing 360 video data and audio data from a plurality of viewpoints within a real-world environment. The method may also include preprocessing and compressing the 360 video data and the audio data into a three-dimensional representation suitable for display. The method may further include rendering a virtual environment of the real-world environment. In addition, the method may include creating a blended virtual environment by combining the captured 360 video data and the audio data with the rendered virtual environment. Further, the method may include displaying the blended virtual environment in a display apparatus of a user.