Neural Network Patch Blending for 6DoF Video Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rendering of six-degree of freedom (6DoF) video is computationally expensive and requires large storage and transmission capacities due to the complexity of point cloud data, and the seamless blending of patches in 6DoF video is a difficult and costly process.
Innovation Solution
A neural network-based approach for patch blending is employed, where full image data is used as ground truth to train a convolutional neural network for seamless image blending, reducing the computational resources required for patch blending.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If point cloud data is used to represent 6DoF video, then immersive video quality is improved, but computational cost and storage requirements increase significantly
Solution Approach 1:
The video content is divided into multiple patches, each representing a specific region of interest. Instead of processing the entire point cloud data, the system segments the scene into discrete patches that can be independently rendered and composited, significantly reducing computational complexity while maintaining visual quality.
Solution Approach 2:
The patent uses 2D image patches as simplified representations (copies) of the complex 3D point cloud data. These 2D patches are generated from the point cloud but contain only the essential visual information needed for rendering, eliminating the need to process and transmit the full point cloud dataset.
2Area of stationary object
If multiple camera patches are composed to form seamless video, then coverage of visible content is improved, but blending complexity and processing cost increase
Solution Approach 1:
The patent introduces an intermediary blending process that uses learned transformation models to smoothly transition between adjacent camera patches. Instead of directly compositing raw patches, the system applies intermediate blending operations that account for geometric distortions and lighting variations, creating seamless transitions without requiring complex real-time processing.
Solution Approach 2:
The blending parameters and transformation models are pre-computed during the encoding phase. This preliminary action allows the client device to simply apply pre-determined blending operations during playback, rather than performing complex real-time blending calculations, significantly reducing processing requirements.
3Reliability
If traditional patch blending methods are used, then seamless composition is achieved, but processing time and computational resources increase
Solution Approach 1:
The patent transforms the blending problem from a complex pixel-level operation into a parameter-based transformation problem. By representing patches with learned geometric and photometric parameters, the system can apply simple parameter transformations instead of expensive pixel-wise blending operations, maintaining seamless composition while dramatically reducing processing time.
Solution Approach 2:
The patent replaces traditional mechanical blending algorithms (such as Poisson blending or gradient domain methods) with a learned transformation model. This substitution uses data-driven approaches to predict blending parameters, eliminating the need for iterative numerical optimization and significantly accelerating the blending process.
Data Source
AI summary
Embodiments are generally directed to neural network based patch blending for immersive video. An embodiment of a system includes one or more processor cores; a memory to store data for images in immersive video; and a neural network training framework. The neural network training framework is to generate a trained neural network for blending of a view of an object with patches generated for one or more detected occlusions for the object.