Virtual Viewpoint Image Generation with Adaptive Coloring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating virtual viewpoint images from multiple camera captures face challenges in accurately creating three-dimensional models, especially for subjects with recessed shapes, leading to reduced image quality due to low shape estimation accuracy and inappropriate coloring processes.
Innovation Solution
An image generation apparatus that evaluates shape data from multiple camera captures to select an appropriate method for determining pixel values in virtual viewpoint images, ensuring accurate representation by adjusting the coloring method based on the accuracy of the three-dimensional model, potentially using a single camera's image for points with low accuracy to prevent color discrepancies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If Visual Hull method is used to generate three-dimensional model, then the model can be created from multiple camera images, but the shape accuracy is reduced for subjects with recessed shapes
Solution Approach 1:
The patent changes the parameter of shape estimation accuracy by introducing a specific evaluation metric that detects recessed shapes. When recessed shapes are detected, the system switches to an alternative coloring method that compensates for the Visual Hull method's inability to accurately represent recessed areas, thereby maintaining manufacturing precision without sacrificing ease of model generation.
2Loss of information
If coloring process is performed using images from multiple cameras, then color information can be obtained for three-dimensional model points, but image quality is reduced when shape estimation accuracy is low
Solution Approach 1:
The patent applies local quality by evaluating each point on the three-dimensional model individually to determine if it lies within a recessed shape. For points in recessed areas, a different coloring method is applied compared to points on convex surfaces. This localized approach ensures that color information is obtained appropriately for each region, maintaining image quality without sacrificing color information completeness.
3Device complexity
If a single coloring method is used for all three-dimensional model points, then the processing is simple, but image quality deteriorates when recessed shapes are present
Solution Approach 1:
The patent introduces dynamics by making the coloring method adaptive rather than static. The system dynamically selects between different coloring methods based on the local geometric characteristics of each point on the three-dimensional model. This dynamic approach allows the system to maintain simple processing for convex surfaces while automatically applying more sophisticated methods for recessed areas, thereby improving image quality without excessive complexity.
Data Source
AI summary
An image generation apparatus that is configured to generate a virtual viewpoint image comprises: an obtaining unit configured to obtain shape data indicating a three-dimensional shape of an object captured from a plurality of directions by a plurality of cameras; a selecting unit configured to select, based on a result of an evaluation on the shape data obtained by the obtaining unit, a method of determining a pixel value in the virtual viewpoint image from among a plurality of methods; and a generating unit configured to generate a virtual viewpoint image by the method of determining the pixel value selected by the selecting unit, wherein the virtual viewpoint image based on images obtained based on capturing by the plurality of cameras, the shape data obtained by the obtaining unit, and viewpoint information representing a virtual viewpoint.