Virtual-Viewpoint Image Generation Using Boundary-Based Pixel Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for generating virtual-viewpoint images from multiple camera captures often result in image degradation due to errors in object shape and camera parameters, leading to unnatural colors, especially at object outlines.
Innovation Solution
An image processing apparatus that generates depth images, detects object boundaries, and creates a pixel map to determine the contributing ratio for each pixel, thereby generating a virtual-viewpoint image that suppresses image degradation by referencing pixels within the object and separated from edges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If texture mapping is performed using reference images with high weighting based on camera position and line of sight, then the virtual-viewpoint image generation speed is improved, but image degradation occurs at object outlines due to errors in three-dimensional model positioning
Solution Approach 1:
The patent segments the image into two distinct regions: a first region corresponding to the inside of the object and a second region corresponding to the outline of the object. This segmentation allows different processing methods to be applied to each region, thereby resolving the contradiction between generation speed and outline quality by treating these regions differently.
Solution Approach 2:
The patent applies different quality standards and processing methods to different parts of the image. For the first region (object interior), texture mapping with high weighting is used for speed. For the second region (object outline), a different approach referencing multiple captured images is used to ensure precision. This local differentiation resolves the contradiction by optimizing each region according to its specific requirements.
2Productivity
If the three-dimensional model is generated from multiple cameras with known parameters, then the virtual-viewpoint image can be generated efficiently, but errors in camera parameters and object shape cause pixel misalignment and unnatural colors
Solution Approach 1:
The patent separates the processing of object interior pixels from outline pixels. By identifying pixels in the outline region through boundary detection, the system can apply more precise processing methods specifically to these pixels without affecting the overall generation efficiency for the entire image.
Solution Approach 2:
The patent changes the weighting parameters differently for different regions. For outline pixels in the second region, instead of using high weighting from a single reference image, the system references multiple captured images with adjusted weighting to compensate for parameter errors, thereby improving pixel positioning accuracy while maintaining overall efficiency.
Data Source
AI summary
The invention comprises a depth-image generator which, based on shape information representing a shape of an object in the captured images of the cameras, generates depth images corresponding to the captured images obtained from a plurality of cameras; a detector which, based on a generated depth image, detects a boundary region of the object in a corresponding captured image; a pixel map generator which, based on the boundary region in the captured image detected by the detector, generates a pixel map representing a contributing ratio for generating the virtual-viewpoint image for each pixel position of the captured image; and an output-image generator which generates the virtual-viewpoint image based on the captured images obtained from the plurality of cameras, and the pixel map corresponding to the captured images.