Virtual Viewpoint Imaging for Transparent Subject Color Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing techniques fail to generate virtual viewpoint images of subjects with high transmittance without incorporating background textures from different viewpoints, leading to a sense of incongruity.
Innovation Solution
An image processing system that generates virtual viewpoint images by separating transparent and opaque portions of a subject using viewpoint information, and combines these images to remove background colors, ensuring consistent color information across different viewpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the texture of the image including the background in the real space viewed from the image capturing apparatus is generated in the area of the subject in the virtual viewpoint image, then the virtual viewpoint image includes background information, but a sense of incongruity is generated when the virtual viewpoint position differs from the image capturing apparatus position
Solution Approach 1:
The patent segments the subject area from the background area in captured images. It identifies and extracts only the subject portion (excluding background) from each captured image, then uses these segmented subject areas to generate the virtual viewpoint image. This segmentation prevents background information from being incorrectly transferred to the virtual viewpoint image, resolving the incongruity issue while preserving necessary subject information.
Solution Approach 2:
The patent extracts and removes background information from the captured images before generating the virtual viewpoint image. By taking out the background component and retaining only the subject portion, the system ensures that the virtual viewpoint image contains accurate subject representation without contaminated background textures from different viewpoints, thereby maintaining visual consistency.
2Reliability
If background color information is removed from the subject area in captured images, then visual consistency is improved, but color information about other subjects behind the high transmittance subject cannot be preserved
Solution Approach 1:
The patent performs preliminary segmentation to identify and separate different subject areas before generating the virtual viewpoint image. By预先 identifying which areas correspond to different subjects and their transmittance characteristics, the system can selectively handle color information - removing background colors from high transmittance subjects while preserving color information of other subjects that are intentionally visible through them.
Solution Approach 2:
The patent applies different processing strategies to different regions of the image based on local characteristics. For areas with high transmittance subjects, it removes background color information to maintain consistency. For areas where other subjects are visible through the high transmittance subject, it preserves the color information of those visible subjects. This local differentiation resolves the contradiction by applying appropriate color handling to each region.
Data Source
AI summary
An image processing apparatus includes an obtaining unit configured to obtain viewpoint information indicating a position of a virtual viewpoint and a line-of-sight direction from the virtual viewpoint, a first generation unit configured to generate a first virtual viewpoint image including a transparent or translucent first portion of a subject using the viewpoint information and a first captured image including the first portion and a pixel not corresponding to an opaque second portion of the subject to be captured through the first portion among pixels corresponding to the first portion, and generate a second virtual viewpoint image including the second portion using the viewpoint information and a second captured image including the second portion, and a second generation unit configured to generate a third virtual viewpoint image including the first portion and the second portion based on the first virtual viewpoint image and the second virtual viewpoint image.