Volumetric Video Encoding via 2D Projection Planes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current volumetric video coding technologies face inefficiencies in compressing dynamic 3D scenes due to changes in geometry and attributes, leading to poor spatial and temporal coding performance, especially in representing and encoding large-scale 3D dynamic maps.
Innovation Solution
The method involves projecting three-dimensional scenes onto simple geometric surfaces like spheres, cylinders, or planes, unfolding these surfaces into 2D planes, and applying standard 2D video coding techniques to the texture and geometry information, with relevant projection geometry details transmitted separately for efficient encoding and decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If volumetric video data is represented using triangle meshes, point clouds, or voxel arrays to enable 6DOF viewing capabilities, then the adaptability for VR/AR/MR applications is improved, but the spatial and temporal coding performance deteriorates
Solution Approach 1:
The patent segments the 3D volumetric video data into multiple 2D projection planes (e.g., front, back, left, right, top, bottom views). Each projection plane is encoded independently using standard 2D video coding techniques, which significantly improves coding efficiency while preserving the ability to reconstruct views from any viewpoint for 6DOF applications.
Solution Approach 2:
The patent transforms the 3D volumetric data into 2D projection planes, effectively reducing dimensionality for encoding purposes. This dimensionality change allows the use of efficient 2D video coding standards while maintaining the capability to synthesize 3D views through reprojection algorithms at the decoder.
2Productivity
If 2D-video based approaches (multiview + depth) are used for compressing volumetric data, then the compression efficiency is improved, but the coverage of the full scene and 6DOF capabilities deteriorate
Solution Approach 1:
The patent creates multiple 2D projection planes that serve dual purposes: they enable efficient compression through standard 2D video coding while simultaneously providing the geometric information needed for 6DOF view synthesis. The multi-planar representation is universally applicable to both compression and flexible viewing requirements.
Solution Approach 2:
The patent uses 2D projection planes as an intermediary representation between the original 3D volumetric data and the final decoded output. These projection planes act as a compact intermediate form that facilitates efficient compression while preserving all necessary information for reconstructing arbitrary 3D views through reprojection.
3Adaptability or versatility
If the geometry and attributes of 3D scenes change dynamically, then the adaptability to dynamic content is improved, but the efficiency of motion-compensation and correspondence identification deteriorates
Solution Approach 1:
The patent segments dynamic 3D scenes into multiple static 2D projection planes for each time instance. This segmentation eliminates the need for complex 3D motion compensation by allowing independent 2D motion estimation on each projection plane, significantly improving efficiency while maintaining adaptability to dynamic content through frame-by-frame or key-frame encoding.
Data Source
Figure 1
Figure 2a~2b
Figure 3a~3b
AI summary
There are disclosed various methods, apparatuses and computer program products for volumetric video encoding and decoding. In some embodiments,two or more patches formed from a three-dimensional image information are obtained, each patch representing projection data of at least a part of an object to a projection plane. A rectangle totally covering the patch is determining for each of the two or more patches. A sorting criteria is determined on the basis of a width and a height of the rectangle. The patches are sorted on the basis of the sorting criteria of the rectangles determined for the two or more patches. An initial size of a grid is selected on the basis of one or more of the largest rectangles; and the two or more patches are inserted into the grid. The grid is encoded into a bitstream.