Extended Reality Recording With 3D Reconstruction for Viewpoint Playback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies fail to capture and playback extended reality (XR) experiences from multiple viewpoints, leading to viewer discomfort due to limited perspective and lack of three-dimensional recording capabilities.
Innovation Solution
A system and method for recording XR experiences that includes capturing time-ordered three-dimensional information, video, and audio, along with computer-generated content, and generating a three-dimensional reconstruction, allowing playback from any viewpoint by aligning this data with a viewer's perspective.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional video recording is used, then the recording process is simple, but the playback is limited to a single fixed viewpoint
Solution Approach 1:
The recording system is divided into multiple independent camera units, each capturing video from its own viewpoint. This segmentation allows the system to record from multiple perspectives simultaneously without requiring a single complex recording device, thereby achieving viewpoint flexibility while managing device complexity through modular design
Solution Approach 2:
The patent transitions from traditional two-dimensional video recording to three-dimensional spatial recording by adding viewpoint dimensionality. Multiple cameras are positioned at different spatial locations to capture video from multiple perspectives, enabling playback from any viewpoint by reconstructing the three-dimensional scene geometry
2Adaptability or versatility
If three-dimensional recording is implemented, then multiple viewpoints are enabled, but the recording and processing complexity increases
Solution Approach 1:
The system performs preliminary actions during recording by pre-calculating and storing camera calibration data, intrinsics, and extrinsics parameters for each camera unit. This preliminary preparation simplifies the subsequent playback process, as the geometric relationships and transformation matrices are already computed and stored, reducing the processing complexity during playback
Solution Approach 2:
Instead of capturing a single monolithic video stream, the system creates multiple independent video copies from different camera perspectives. Each camera unit produces its own video stream that can be independently processed and stored, simplifying the overall data management and processing complexity while enabling multi-viewpoint playback through reconstruction
3Ease of operation
If real-time rendering is used, then the experience is immersive, but the computational resources required are substantial
Solution Approach 1:
The system performs preliminary rendering and processing during the recording phase, pre-computing the three-dimensional scene representation and storing it in an optimized format. This preliminary action reduces the computational burden during playback, as the heavy lifting is done beforehand, thereby maintaining user immersion while reducing real-time energy consumption
Solution Approach 2:
The patent employs a rendering engine that can be reset and reconfigured for different playback scenarios without requiring persistent complex computational resources. The system uses efficient data structures and algorithms that can be quickly loaded and processed, reducing the sustained computational energy consumption while maintaining high-quality immersive experience
Data Source
AI summary
Implementations of the subject technology provide systems and methods for recording an extended reality experience in a way that allows the experience to be played back at a later time from a different viewpoint or perspective. This allows computer-generated content that was rendered for display to a user during the recording, to be re-rendered during playback at the correct time and location in the recording, but from a different perspective. In order to facilitate this type of viewer-centric playback, the recording includes a computer-generated content track that references resources for re-rendering the computer-generated content at each point in time in the recording.