Virtual Viewpoint Synthesis With Selective 3D Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating virtual viewpoint images require significant hardware resources and numerous operations due to the processing involved in projecting images to a three-dimensional model, such as projective transformation and compositing, which is computationally intensive.
Innovation Solution
An image processing system that includes capturing apparatuses and a server apparatus, where the capturing apparatuses preprocess images by separating foreground and background, and the server apparatus manages background models and coordinates to efficiently project images to meshes of the model, reducing the need for extensive processing by deciding and transmitting only specific regions of images to be projected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If projective transformation and compositing processing is performed on captured images to project them to a three-dimensional model, then a virtual viewpoint image can be generated, but hardware resources (memories) and computational operations are significantly consumed
Solution Approach 1:
The patent divides the captured image into multiple regions (first region and second region) based on depth information and foreground/background separation. Only the first region (background portion) is projected to the three-dimensional model, while the second region (foreground portion) is extracted and composited separately. This segmentation reduces the amount of image data that requires computationally intensive projective transformation and compositing operations, thereby reducing hardware resource consumption while maintaining virtual viewpoint image generation quality.
Solution Approach 2:
The patent extracts the foreground portion (second region) from the captured image before performing projective transformation. By taking out the foreground objects that do not require projection to the background model, the system avoids unnecessary computational operations on these regions. The extracted foreground is then composited with the projected background to form the final virtual viewpoint image, significantly reducing the computational load compared to processing the entire image.
2Reliability
If projective transformation and compositing processing is performed on captured images, then a virtual viewpoint image can be generated, but the processing requires many separate operations and computational steps
Solution Approach 1:
The patent segments the image processing into distinct stages: foreground/background separation, region division, selective projection of only the first region to the three-dimensional model, and compositing of the second region with the projected background. This segmented approach allows each processing stage to operate on optimized data subsets, reducing the total number of computational operations required compared to traditional methods that process entire images through all processing stages.
Solution Approach 2:
The patent applies partial action by performing projective transformation only on the first region (background portion) that actually requires projection to the three-dimensional model, rather than applying the transformation to the entire captured image. This partial processing approach maintains the quality of the virtual viewpoint image while significantly reducing the number of computational operations required, as the second region (foreground) is handled through simpler extraction and compositing operations.
Data Source
Figure 1
Figure 2A~2B
Figure 3A~3B
AI summary
An information processing apparatus comprises: a capturing information obtaining unit for obtaining capturing information on plural capturing apparatuses for capturing images in plural directions; a model information obtaining unit for obtaining model information on a background shape model indicating a three-dimensional shape of a background of a virtual viewpoint image generated by an image generating unit using the images captured by the capturing apparatuses; and an information generating unit for generating related information associating at least one of the capturing apparatuses with a partial region of the background shape model, based on the obtained capturing information and the obtained model information, wherein the related information is used to determine which one of the capturing apparatuses is to capture the image used by the image generating unit to decide a pixel value corresponding to the partial region in the virtual viewpoint image.