Immersive Video Patch Segmentation for Accurate Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current MPEG Immersive Video (MIV) standards struggle with encoding and decoding volumetric videos where texture and depth components are captured by separate devices, leading to inefficiencies in rendering and increased dizziness due to inconsistent visual feedback during head translations.
Innovation Solution
Incorporating segmentation information in the encoding process to associate regions of source views with entities, allowing for efficient patch-based encoding and rendering, including flags to indicate the source of segmentation information, whether based on depth or texture images, and performing warping and blending operations accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If texture and depth components are captured by separate devices, then flexibility in capturing is improved, but rendering accuracy deteriorates due to misalignment
Solution Approach 1:
The patent introduces segmentation information as an intermediary element that bridges texture and depth components captured by separate devices. By segmenting the scene into distinct objects and tracking their motion independently, the system can align texture and depth data from different capture devices, resolving the misalignment issue while maintaining the flexibility of separate device capture
Solution Approach 2:
The patent applies segmentation by dividing the captured scene into multiple discrete objects or regions. Each segmented region is associated with motion information that enables precise alignment between texture and depth components from separate capture devices, thereby improving rendering accuracy without sacrificing capturing flexibility
2Manufacturing precision
If segmentation information is incorporated in encoding, then rendering accuracy is improved, but device complexity increases
Solution Approach 1:
The patent performs segmentation and generates motion information during the encoding phase, preparing all necessary alignment data in advance. This preliminary action allows the decoder to simply retrieve and apply pre-computed segmentation and motion information, improving rendering accuracy without significantly increasing decoding complexity
Solution Approach 2:
The patent creates simplified representations of scene objects through segmentation masks and motion vectors that can be easily stored and transmitted. These copied representations (segmentation information and motion data) enable accurate rendering without requiring the full complexity of the original multi-device capture system
3Device complexity
If inconsistent visual feedback is provided during head translations, then device simplicity is maintained, but user comfort deteriorates due to dizziness
Solution Approach 1:
The patent implements a feedback mechanism where motion information derived from segmentation is used to dynamically adjust the rendered view during head translations. This feedback loop ensures that the visual output remains consistent with the user's head movements, preventing dizziness while maintaining relative system simplicity by using pre-computed motion data
Data Source
AI summary
Systems and methods are provided in which a plurality of source views of a scene are encoded as an immersive video comprising a plurality of patches. The patches are segmented based on segmentation information from at least one of the source views. The segmentation information associates each of a plurality of regions of the source view with a respective entity. Information is also encoded that indicates which of the source views are associated with the segmentation information used to segment the patches. This information may be used to provide improved rendering, particularly where rendering is performed of less than all entities in the scene.