Virtual Viewpoint Image Generation Using Shadow Mask Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual viewpoint image generation techniques face increased data requirements with higher image resolution or the number of imaging apparatuses, particularly in shadow rendering, leading to inefficient data processing and unnatural color changes.
Innovation Solution
An image processing apparatus generates shadow mask images to extract shadow areas from foreground images, removes shadows to create non-shadow foreground images, and renders shadows in the background images based on virtual viewpoint information, reducing data complexity and improving rendering quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the resolution of images or the number of imaging apparatuses is increased, then the quality of virtual viewpoint image is improved, but the amount of data required for generating the virtual viewpoint image increases
Solution Approach 1:
The patent segments the image processing into distinct components: foreground image processing, background image processing, and shadow processing. By separating shadow extraction into an independent stage using shadow mask images, the system can process each component independently and efficiently, reducing overall data processing requirements while maintaining high image quality even with increased resolution or number of imaging apparatuses
Solution Approach 2:
The patent extracts shadow information from foreground images to create separate shadow mask images. This extraction allows the shadow data to be processed and rendered independently from the main foreground and background images, reducing the data burden on the main processing pipeline while preserving shadow details for high-quality virtual viewpoint generation
2Reliability
If shadow rendering is performed using foreground images and background images, then the naturalness of shadows is improved, but the data processing complexity increases
Solution Approach 1:
The patent divides shadow rendering into separate processing stages: shadow extraction from foreground images, shadow mask generation, and shadow composition with background images. This segmentation reduces processing complexity by handling shadow data independently rather than processing complete foreground and background images together, while still achieving natural shadow effects through dedicated shadow processing pipelines
Data Source
AI summary
A virtual viewpoint foreground image generating unit generates a virtual viewpoint foreground image, which is an image of a foreground object seen from a virtual viewpoint without a shadow, based on received multi-viewpoint images and a received virtual viewpoint parameter. A virtual viewpoint background image generating unit generates a virtual viewpoint background image, which is an image of a background object seen from the virtual viewpoint, based on the received multi-viewpoint images and virtual viewpoint parameter. A shadow mask image generating unit generates shadow mask images from the received multi-viewpoint images. A shadow-added virtual viewpoint background image generating unit renders a shadow in the virtual viewpoint background image based on the received virtual viewpoint background image, shadow mask images, and virtual viewpoint parameter. A combined image generating unit generates a virtual viewpoint image by combining the virtual viewpoint foreground image with the shadow-added virtual viewpoint background image.