Virtual Viewpoint Rendering with Boundary-Aware Object Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating virtual viewpoint images face issues such as decreased resolution and incomplete shape representation when objects are positioned near the boundary of the imaging area, leading to a sense of unnaturalness and reduced image quality.
Innovation Solution
An information processing apparatus that generates virtual viewpoint images by correcting object portions based on their positional relationship with the imaging area, using methods like shading or transparency processing to enhance image quality and completeness, particularly at area boundaries and for visible backfaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a virtual viewpoint image is generated using a three-dimensional model obtained from multiple imaging positions, then the image can represent the object from different viewpoints, but the resolution decreases when the distance between the virtual viewpoint and the three-dimensional model is too short
Solution Approach 1:
The patent applies local quality by differentiating treatment between front-facing surfaces and back-facing surfaces of the three-dimensional model. Front-facing surfaces (those visible from the virtual viewpoint) are rendered with high quality and full resolution, while back-facing surfaces are rendered with lower quality or transparency. This selective quality allocation maintains high resolution for the visible object portions while avoiding the resolution degradation that would occur if the entire model was rendered at high quality from close virtual viewpoints.
2Ease of manufacture
If the three-dimensional model is generated from images captured in a limited imaging area, then the processing can be simplified, but the shape representation becomes incomplete when the object is positioned near the boundary of the imaging area
Solution Approach 1:
The patent determines the visibility of each surface portion of the three-dimensional model from the virtual viewpoint and applies different rendering qualities accordingly. For portions that are visible (front-facing), high quality rendering is applied to maintain shape completeness. For portions that are not visible (back-facing), lower quality or transparent rendering is applied. This allows the system to work with a limited imaging area while maintaining shape completeness for the visible portions of the object.
Solution Approach 2:
Instead of trying to capture all surfaces of the object from multiple imaging positions (which would require a large imaging area), the patent inverts the approach by using a limited imaging area to capture only the necessary portions and then using three-dimensional model rendering to fill in the rest. The visibility determination algorithm effectively inverts the traditional photogrammetry approach by determining which portions need to be rendered based on virtual viewpoint geometry rather than which portions were captured based on physical camera positions.
3Shape
If backface portions of the three-dimensional model are rendered with full opacity, then the image appears complete, but it creates an unnatural appearance when viewed from certain angles
Solution Approach 1:
The patent changes the visual properties (transparency/opacity) of back-facing surfaces to create a more natural appearance. By rendering back-facing surfaces with transparency or reduced opacity, the image mimics real-world visual properties where surfaces not visible from the current viewpoint should appear transparent or faded. This prevents the unnatural appearance that would result from rendering all surfaces with full opacity, while still maintaining shape completeness through the three-dimensional model structure.
Data Source
AI summary
An information processing apparatus is provided. A three-dimensional model of an object, which has been obtained by capturing an image of the object in an imaging area from a plurality of positions, and information indicating a position of the object with respect to the imaging area are obtained. A virtual viewpoint image that includes the object is generated based on the three-dimensional model of the object through performing correction that accords with a position of the object, on the object in the virtual viewpoint image.