Volumetric Video Depth-Color Alignment Using Selective Depth Warping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for encoding and transcoding multi-view plus depth (MVD) images with unaligned color and depth sensors lack techniques to efficiently encode assistance metadata, leading to increased computational burden and memory footprint due to excessive depth map warping operations.
Innovation Solution
Encoding MVD images with assistance metadata that identify a subset of depth views contributing to each color view, reducing the number of depth map warping operations and computational demands by providing decoder-side optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If all depth views are used for view synthesis, then the quality of synthesized views is improved, but the computational burden and memory footprint increase significantly
Solution Approach 1:
The patent segments the set of depth views into multiple subsets, where each subset is associated with a specific color view. Instead of using all depth views for every color view, the system selectively applies warping operations only to the relevant subset, thereby reducing computational complexity while maintaining synthesis quality.
Solution Approach 2:
The patent applies partial action by using only a subset of depth views rather than all available depth views. This selective approach performs sufficient warping operations to achieve acceptable synthesis quality without the excessive computational cost of processing all depth views, optimizing the balance between quality and complexity.
2Loss of information
If depth map warping operations are performed for all depth views, then complete depth information is available for synthesis, but the number of operations increases leading to higher complexity
Solution Approach 1:
The patent segments depth views into subsets and performs warping operations only on the relevant subset for each color view. This segmentation maintains necessary depth information completeness while avoiding redundant warping operations, thereby improving transcoding efficiency.
Solution Approach 2:
The patent extracts and identifies a specific subset of depth views that are most relevant for each color view. By taking out only the necessary depth views from the complete set, the system performs warping operations selectively, reducing the total number of operations while preserving essential depth information.
3Adaptability or versatility
If unaligned MVD images are converted to aligned MVD images, then view synthesis compatibility is improved, but the number of required depth map warping operations increases
Solution Approach 1:
The patent segments the conversion process by creating subsets of depth views for each color view. This segmentation enables the system to achieve alignment compatibility selectively, performing warping operations only where necessary, thereby reducing the overall complexity of the conversion process.
Solution Approach 2:
The patent applies partial action by performing warping operations only on a subset of depth views rather than all depth views. This approach provides sufficient alignment compatibility for view synthesis without the excessive complexity of converting all depth views, optimizing the trade-off between compatibility and complexity.
Data Source
AI summary
Methods, device and data stream are provided to encode unaligned multi-view plus depth images associated with assistance metadata for generating depth maps aligned with the unaligned color views. At the encoding stage, for an unaligned color view, the contribution of each depth map is evaluated. A subset of the most contributing depth maps is selected and metadata representative of this subset is encoded in the data stream of the unaligned multi-view plus depth images in association with the unaligned color view. At the decoding stage, to generate a depth map unaligned with the unaligned color view, only the depth maps referenced in the assistance metadata are warped.