Virtual Viewpoint Image Correction Using 3D Shape Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual viewpoint image generation systems often produce strange or low-quality images due to incorrect background extraction, especially when foreground objects are reflected in dynamic backgrounds, leading to a feeling of strangeness in the generated images.
Innovation Solution
An image processing apparatus that acquires three-dimensional shape data from multiple cameras, corrects pixel values in images based on camera positions and orientations, and generates background texture images by removing foreground objects from background images using a correction unit, thereby preventing residual images from being reflected in the virtual viewpoint image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If foreground and background segmentation processing is performed using motion detection, then moving objects can be extracted as foreground images, but objects in the background with motion (such as spectators' stands) cannot be correctly identified as background, leading to incorrect segmentation
Solution Approach 1:
The patent divides the image processing into two distinct stages: first extracting foreground objects using motion detection, then separately processing background images by removing foreground components. This segmentation allows each stage to focus on specific tasks, improving overall accuracy while managing complexity through modular processing steps.
Solution Approach 2:
The patent extracts foreground objects from the original images using motion detection, then separately extracts and processes the background by removing the foreground components. This extraction approach enables independent optimization of foreground and background processing, resolving the contradiction between accuracy and complexity.
2Productivity
If background images are generated without removing foreground objects, then processing is simpler and faster, but the generated virtual viewpoint images contain residual images of foreground objects reflected on background surfaces, causing a feeling of strangeness
Solution Approach 1:
The patent performs foreground removal from background images as a preliminary step before generating the virtual viewpoint image. By preparing clean background images in advance with foreground objects removed, the system ensures high image quality without compromising generation speed, as the removal process is completed beforehand rather than during real-time rendering.
Solution Approach 2:
The patent applies preliminary anti-action by proactively removing foreground objects from background images before they can cause harmful effects (residual reflections) in the final virtual viewpoint image. This preventive measure eliminates the quality issue while maintaining processing efficiency through batch preparation.
3Measurement precision
If multiple cameras are used to capture images from different positions, then three-dimensional shape data can be generated for more accurate object modeling, but the system complexity and data processing requirements increase significantly
Solution Approach 1:
The patent merges data from multiple cameras to generate three-dimensional shape information, then combines this 3D data with two-dimensional image data in a unified processing framework. This merging approach leverages the advantages of multi-camera systems for accurate 3D modeling while integrating the process efficiently with background processing to manage overall system complexity.
Data Source
AI summary
An image processing apparatus for generating a virtual viewpoint image that acquires a parameter for identifying a position and orientation of a first imaging device, acquires three-dimensional shape data of an object that is generated based on a plurality of images acquired by a plurality of second imaging devices different from the first imaging device, and corrects a pixel value of a pixel included in a region corresponding to the object in an image acquired by the first imaging device, based on the position and orientation of the first imaging device identified based on the acquired parameter and the acquired three-dimensional shape data of the object.