3D View Synthesis Using Mesh-Guided Rendering Beyond the Sweet Spot
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing immersive video systems suffer from limited viewing spaces and significant quality degradation when viewers move outside the sweet spot, due to insufficient 3D data for view synthesis, leading to errors, artefacts, and a suboptimal user experience.
Innovation Solution
A video rendering apparatus that uses a combination of captured video data and a three-dimensional mesh model to generate images for different viewpoints, adaptively blending data from both sources based on the deviation of the viewing pose relative to the capture pose, ensuring consistent image quality and immersive experience across a wider range of movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple cameras are used to capture 3D representation of a scene, then the viewing quality inside the sweet spot zone is improved, but the viewing quality outside the sweet spot zone deteriorates due to missing information and 3D estimation errors
Solution Approach 1:
The patent divides the scene into two distinct data representations: a complete 3D mesh model for the entire scene and captured video data for specific regions. This segmentation allows different data types to be used in different spatial zones, with the mesh providing structural information for areas outside the sweet spot and video data providing high-quality visual information within the sweet spot region.
Solution Approach 2:
The patent creates a composite rendering system that combines two different data sources: a 3D mesh model with texture maps and captured video data. This composite approach allows the system to leverage the geometric accuracy of the mesh model for positioning and the visual fidelity of video data for the sweet spot region, while using the mesh model to fill in areas outside the sweet spot where video data is unavailable.
2Ease of operation
If the viewer moves outside the viewing space, then the freedom of movement is improved, but the image quality deteriorates due to synthesis errors and artefacts
Solution Approach 1:
The patent performs preliminary action by pre-capturing video data within the sweet spot region and pre-processing it to identify regions where video data is available versus where it is not. This pre-processing creates a foundation that allows the system to quickly determine whether to use video data or mesh model data during rendering, enabling smooth transitions as the viewer moves in and out of the sweet spot zone without quality degradation.
3Measurement precision
If captured video data is used for view synthesis, then the image quality within capture pose region is improved, but data communication requirements increase
Solution Approach 1:
The patent applies local quality by using high-fidelity video data only in the specific region where it is most beneficial - the sweet spot or capture pose region - while using the more compact mesh model representation for the rest of the scene. This localized use of video data maximizes image quality where viewers are most likely to be positioned while minimizing overall data transmission requirements.
Data Source
AI summary
An apparatus comprises a receiver (601) receiving captured video data for a real world scene and being linked with a capture pose region. A store (615) stores a 3D mesh model of the real world scene. A renderer (605) generates an output image for a viewport for a viewing pose. The renderer (605) comprises a first circuit (607) arranged to generate first image data for the output image by projection of captured video data to the viewing pose and second circuit (609) arranged to determine second image data for a first region of the output image in response to the three-dimensional mesh model. A third circuit (611) generates the output image to include at least some of the first image data and to include the second image data for the first region. A fourth circuit (613) determines the first region based on a deviation of the viewing pose relative to the capture pose region.