Neural View Synthesis for Wider 3D Pose Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image synthesis methods for immersive video, such as 6DoF and MIV, face challenges in generating high-quality views for different poses due to limited capture data, imperfect depth maps, and complex processing, leading to errors, artefacts, and increased data requirements, which restrict user movement and degrade the immersive experience.
Innovation Solution
A neural network-based approach that utilizes a view synthesis neural network trained with multi-view images and three-dimensional spatial data, independent of view poses, to generate high-quality images for various viewing positions, reducing complexity and data requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional view synthesis methods are used with multi-view images and depth maps, then view images can be generated for different poses, but image quality deteriorates and errors increase when viewpoints deviate from reference viewpoints
Solution Approach 1:
The patent transforms the view synthesis problem from geometric warping operations to a neural network prediction task. By changing the approach from deterministic geometric transformation to probabilistic neural network inference, the system can generalize to viewpoints beyond the training range, improving both quality and adaptability simultaneously.
Solution Approach 2:
The patent replaces the traditional mechanical view synthesis pipeline (depth map generation, view warping, de-occlusion handling) with a neural network-based system. This substitution eliminates the need for explicit depth maps and complex warping operations, allowing high-quality synthesis for a broader range of viewpoints without the limitations of geometric methods.
2Reliability
If immersive video data is provided in multi-view with depth map representation, then three-dimensional scene information is available, but data requirements increase and processing complexity increases
Solution Approach 1:
The patent extracts only the essential three-dimensional spatial information needed for view synthesis while discarding the complex depth map representation. The neural network learns to infer depth and spatial relationships directly from multi-view images, eliminating the need to process separate depth maps and reducing overall system complexity.
Solution Approach 2:
The patent merges the view synthesis and de-occlusion functions into a single neural network model. Instead of separately processing depth maps, performing warping operations, and handling de-occlusion, the unified neural network performs all these tasks simultaneously, significantly reducing processing complexity while maintaining reliability.
3Adaptability or versatility
If view synthesis is performed using depth steered rendering and multiple reference views, then new views can be synthesized, but computational complexity increases and processing time increases
Solution Approach 1:
The patent uses a neural network to learn and copy the view synthesis function from training data. Instead of performing complex real-time calculations with multiple reference views and depth steered rendering, the system copies the synthesized view directly from the neural network's learned parameters, dramatically improving processing speed while maintaining versatility.
Solution Approach 2:
The patent performs view synthesis in advance during the neural network training phase. The network learns to predict views for various poses during training, so during actual operation, no complex real-time synthesis is needed. This preliminary action transfers the computational burden from runtime to training time, improving processing speed without sacrificing capability.
Data Source
AI summary
A first apparatus comprises a first receiver (301) receiving images of a scene captured and a second receiver (303) receives 3D spatial data for the scene A view synthesis neural network (307) generates view shifted images for the scene for different view poses from the images and the spatial data. A neural network trainer (309) trains the view synthesis neural network (307) based on images of the scene for different view poses. A generator (305) generates an audiovisual data stream comprising: image data for the images, scene data representing the three dimensional spatial data, and coefficient data describing coefficients of the view synthesis neural network (307) after training. A second apparatus receives the audio visual data stream and sets a local neural network (403) based on the coefficient data. The local neural network (403) is then used to generate images of the scene for different view poses.