Virtual Viewpoint Image Generation with Color Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing techniques for generating virtual viewpoint images from multi-viewpoint images often result in unnatural color changes around the boundaries of areas textured-mapped from different image capturing apparatuses, causing visual oddness due to lighting and equipment differences.
Innovation Solution
An image processing apparatus that determines pixel values for virtual viewpoint images by combining pixel values from different image capturing apparatuses, using a determination unit to calculate pixel values for areas not captured by one apparatus at a specific time based on values from other apparatuses, and generates virtual viewpoint images by transforming and combining background, shadow, and object image data to minimize color discrepancies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Area of stationary object
If texture images from multiple image capturing apparatuses are blended and texture-mapped to generate a virtual viewpoint image, then the coverage and completeness of the virtual viewpoint image are improved, but unnatural color changes appear around the boundaries between areas from different apparatuses
Solution Approach 1:
The patent applies local quality by determining pixel values differently based on local conditions: for pixels in areas captured by the first image capturing apparatus, pixel values are determined based on temporal changes (first time and second time images), while for pixels in areas not captured by the first apparatus, pixel values are determined based on the second image capturing apparatus. This localized approach ensures color consistency within each region while maintaining overall image completeness.
Solution Approach 2:
The patent uses an intermediary approach by introducing a determination unit that selectively chooses which image capturing apparatus's data to use for each pixel. This intermediary decision-making process prevents direct blending of color-different images at boundaries, instead using temporal data from the same apparatus or spatial data from a different apparatus only when necessary, thereby maintaining color consistency.
2Adaptability or versatility
If images from different image capturing apparatuses are combined to create a virtual viewpoint image, then the versatility and viewpoint flexibility are improved, but visual oddness occurs due to lighting and equipment differences between apparatuses
Solution Approach 1:
The patent ensures visual coherence by applying local quality: each pixel's value is determined based on the specific capturing conditions and temporal availability. Pixels are processed differently depending on which apparatus captured them and when, ensuring that lighting and equipment characteristics remain consistent within each region while still achieving versatile viewpoint generation.
Solution Approach 2:
The patent applies preliminary action by pre-determining which pixels should be filled from which image capturing apparatus based on temporal and spatial analysis. The determination unit proactively identifies areas not captured by the first apparatus and prepares to fill them from the second apparatus, preventing visual oddness before it occurs in the final image generation.
3Measurement precision
If pixel values are determined using images from the first image capturing apparatus at different times, then temporal resolution and motion accuracy are improved, but the completeness of the virtual viewpoint image decreases for areas not captured at the first time
Solution Approach 1:
The patent merges data from multiple image capturing apparatuses and multiple time points to create a complete virtual viewpoint image. The determination unit combines temporal data (first time and second time images from the first apparatus) with spatial data (images from the second apparatus) to fill gaps, ensuring both temporal precision and information completeness are achieved.
Solution Approach 2:
The determination unit performs preliminary analysis to identify which pixels are available from temporal data and which require spatial data from other apparatuses. This preliminary action ensures that temporal precision is maintained where possible, while information completeness is restored by selectively incorporating data from the second image capturing apparatus.
Data Source
AI summary
An image processing apparatus includes: an image obtaining unit configured to obtain images of an image capturing target area from a plurality of directions; an information obtaining unit configured to obtain viewpoint information; a determination unit configured to determine a pixel value of a particular area in a virtual viewpoint image according to a virtual viewpoint at a first time indicated by the viewpoint information, based on a pixel value of an obtained image based on image capturing by a first image capturing apparatus at a second time, and a pixel value of an obtained image based on image capturing by a second image capturing apparatus at the first time, wherein the particular area is not captured by the first image capturing apparatus at the first time; and a generation unit configured to generate a virtual viewpoint image in accordance with the determination of the pixel value.