Mesh Geometry Prediction for High Efficiency Mesh Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The compression of mesh representations in 3D video data is challenging due to irregularly sampled vertices and the need for signaling polygons for mesh-based rendering, leading to high computational demands and bandwidth requirements, which are not efficiently addressed by existing technologies.
Innovation Solution
A deep learning-based system using vertex and connectivity prediction networks to generate approximated high-resolution mesh geometry from low-resolution representations, employing Minkowski Networks and Multi-Resolution Convolution Blocks for efficient coding and decoding of mesh geometry.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If mesh geometry is compressed using traditional methods, then storage and transmission requirements are reduced, but coding efficiency is insufficient due to irregularly sampled vertices and complex polygon signaling requirements
Solution Approach 1:
The mesh geometry is divided into two parts: a low-resolution base mesh that captures the overall structure, and a set of vertex displacement vectors that encode the high-resolution details. This segmentation allows efficient compression by separating the coarse structure (easy to transmit) from the fine details (compressed using predictive models).
Solution Approach 2:
A neural network is trained in advance to predict vertex displacements from the low-resolution mesh. During compression, this pre-trained model generates predictions that are then used to encode only the residuals, significantly reducing the information that needs to be transmitted while maintaining high coding efficiency.
2Measurement precision
If high-resolution mesh data is transmitted via communication network, then quality is maintained, but bandwidth requirements become extremely high causing network delays and unacceptable latency
Solution Approach 1:
The method extracts only the essential information needed to reconstruct the high-resolution mesh by representing it as deviations from a low-resolution base mesh. The vertex displacement vectors contain only the necessary correction information, dramatically reducing the data quantity while preserving geometric accuracy.
Solution Approach 2:
Instead of transmitting the complete high-resolution mesh, the system transmits a low-resolution mesh copy plus compact displacement vector data. The decoder reconstructs the high-resolution mesh by applying the displacement vectors to the low-resolution mesh, achieving high accuracy with minimal bandwidth consumption.
3Adaptability or versatility
If vertices are irregularly sampled in 3D space, then mesh representation flexibility is improved, but compression becomes challenging due to the irregular sampling pattern
Solution Approach 1:
The compression method dynamically adapts to irregular vertex sampling by using a neural network that processes the actual vertex positions and connectivity as input. The network learns the specific irregular sampling pattern of the input mesh and generates appropriate displacement predictions, making the compression algorithm flexible to any vertex distribution without requiring complex adaptive logic.
Data Source
AI summary
Systems and methods are provided for efficiently encoding geometry information for 3D media content. An illustrative system generates a low-resolution polygon mesh from a high-resolution polygon mesh. The system uses a vertex occupancy prediction network to generate, from vertices of the low-resolution polygon mesh, approximated vertices of the high-resolution polygon mesh. The system uses a connectivity prediction network to generate, from approximated vertices of the high-resolution polygon mesh, approximated connections of the high-resolution polygon mesh. The system computes vertex errors between the approximated vertices and the vertices of the high-resolution polygon mesh, and connectivity errors between the approximated connections and the connections of the high-resolution polygon mesh. The system transmits, to a receiver over a communication network, bitstreams of the low-resolution polygon mesh, the vertex errors, and the connectivity errors for reconstruction of the high-resolution polygon mesh and display of the 3D media content.