Immersive Video Metadata for In-Painted Patch Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating immersive 6DoF video face challenges in handling missing data regions during view synthesis, as in-painted depth information is of lower quality and can cause conflicts during reconstruction, increasing bitrate and resource requirements.
Innovation Solution
The proposed solution involves encoding multi-view data with metadata that distinguishes between original and in-painted patch data units, using a field to indicate the presence of in-painted data, allowing for controlled rendering priorities and reduced bit- and pixel-rates by employing a Level of Detail (LoD) feature.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If in-painting is performed during data encoding to fill missing image regions, then the completeness of the target view is improved, but the re-projection accuracy deteriorates due to lower quality in-painted depth information
Solution Approach 1:
The patent segments the multi-view data into original captured data and in-painted data, treating them as separate entities with different quality characteristics. This segmentation allows the system to selectively apply different processing and rendering strategies to each type of data, preserving the high quality of original data while still utilizing in-painted data to fill missing regions.
Solution Approach 2:
The patent applies local quality by assigning different rendering priorities to different regions based on their data origin. Original captured data is rendered with high priority to maintain accuracy, while in-painted data is rendered with lower priority. This local differentiation ensures that the most important regions (those with original data) maintain high re-projection accuracy while still filling gaps with lower quality in-painted content.
2Productivity
If in-painted patches are packed with original patches in the same video stream, then the bitrate efficiency is improved, but conflicts arise during view reconstruction when both types of patches map to the same position
Solution Approach 1:
The patent introduces a feedback mechanism through metadata that explicitly identifies which patches contain in-painted data. This metadata provides feedback to the reconstruction process, enabling the system to detect potential conflicts between in-painted and original patches mapping to the same position and resolve them by prioritizing original data over in-painted data.
Solution Approach 2:
The patent uses metadata as an intermediary between the encoded patches and the reconstruction process. This intermediary layer carries information about the origin and quality of each patch, enabling the reconstruction algorithm to make informed decisions about how to combine or select between in-painted and original patches without direct conflict.
3Loss of information
If additional in-painted textures are packed within the video streams, then the completeness of the target view is improved, but the bitrate and frame size increase, placing higher resource requirements on the client device
Solution Approach 1:
The patent applies partial action by selectively including only the necessary in-painted patches in the video stream, rather than encoding all possible in-painted content. The system encodes in-painted patches only for regions that are actually missing from the source views and would benefit from in-painting, thereby reducing the overall bitrate and frame size while still achieving view completeness.
Data Source
AI summary
Concepts for encoding and decoding multi-view data for immersive video are disclosed. In an encoding method, metadata is generated comprising a field indicating if a patch data unit of the multi-view data comprises in-painted data for representing missing data. The generated metadata provides a means of distinguishing patch data units comprising original texture and depth data from patch data units comprising in-painted data (e.g. in-painted texture and depth data). The provision of such information within the metadata of immersive video may address problems associated with blending and pruned view reconstruction. Also provided are an encoder and a decoder for multi-view data for immersive video, and a corresponding bitstream, comprising metadata.