Compact Multiplane Images Using Redundancy-Penalized Layer Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multiplane image (MPI) representations are bulky due to their memory footprint, especially when encoded as atlases, and mechanisms to generate compact MPIs with controlled layers and large transparent areas are lacking.
Innovation Solution
A method involving a trained convolutional neural network to determine redundancy between layers of a multiplane image, using a penalization function to reduce redundant information and generate compact MPIs, and a device for synthesizing views and encoding data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiplane images are represented with multiple layers to enable view synthesis, then view synthesis quality is improved, but memory footprint increases
Solution Approach 1:
The patent extracts and removes redundant information between layers by using a trained convolutional neural network to identify and eliminate duplicate or highly similar pixel data across multiple layers. This extraction of essential information while discarding redundancy directly reduces memory footprint while preserving view synthesis capability.
Solution Approach 2:
The patent applies a penalization function that modifies the optimization parameters during MPI generation. By introducing a redundancy penalty term in the loss function, the system changes the parameter optimization criteria to favor compact representations with fewer redundant layers, thereby reducing memory footprint while maintaining synthesis quality.
2Measurement precision
If all n views are used as input to the neural network, then rendering accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent uses only a subset of n-s views as input to the convolutional neural network rather than all n views. This partial action approach selects the most informative views for input, reducing computational complexity while the trained network compensates to maintain rendering accuracy through learned features from the available subset.
3Productivity
If MPIs are encoded as atlases to reduce data size, then transmission efficiency is improved, but atlas size increases
Solution Approach 1:
The patent extracts and removes redundant information between layers by using a trained convolutional neural network to identify and eliminate duplicate or highly similar pixel data across multiple layers. This extraction of essential information while discarding redundancy directly reduces memory footprint while preserving view synthesis capability.
Solution Approach 2:
The patent applies a penalization function that modifies the optimization parameters during MPI generation. By introducing a redundancy penalty term in the loss function, the system changes the parameter optimization criteria to favor compact representations with fewer redundant layers, thereby reducing memory footprint while maintaining synthesis quality.
Data Source
AI summary
Methods and devices for generated multiplane images from a representation of a 3D scene are disclosed. When an MPI is generated by the system, an image of the accumulation of the alpha values is computed. A convolutional networked is trained with a penalization function that penalizes MPIs with high values of accumulated alpha according to a parameter selected or trained to reduce the redundancy of information between layers of the MPI. When generating a MPI from a subset of the views of a new captured 3D scene, a first MPI is generated with the trained CNN with the associated parameter. An error is calculated between the other views and corresponding synthesized with the first MPI. This error is used to modify the parameter. Then, a second MPI is generated with the trained CNN with the new parameter.