3D Virtual Viewpoint Image Generation Using Depth Segmentation and Inpainting

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Generating accurate three-dimensional (3D) virtual viewpoint images is challenging due to the complexity of estimating consistent depth information across multiple reference viewpoints, which affects the quality of the virtual viewpoint image.

Innovation Solution

A method and apparatus that segment images into layers based on depth information, inpaint occluded areas using reference viewpoint images, and perform image warping to provide parallax, separating and warping background and foreground areas from a central axis movement perspective, ensuring accurate reconstruction and enhanced 3D effect.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If depth information is estimated for all pixels in the scene using multiple reference viewpoint images, then the completeness of the 3D virtual viewpoint image is improved, but the complexity of generating consistent depth information across multiple viewpoints increases

Engineering Contradiction:

Improvecompleteness of depth informationVSAvoidcomplexity of depth information generation

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The image is segmented into multiple layers based on depth information, with each layer representing a different depth range. This segmentation allows the system to process and generate depth information for each layer separately, reducing the overall complexity while maintaining completeness of the 3D virtual viewpoint image.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different processing methods are applied to different regions of the image based on their depth characteristics. Foreground areas, midground areas, and background areas are handled differently, with occluded regions being specifically inpainted using reference viewpoint images. This local differentiation reduces complexity by focusing computational resources where needed.

Inventive Principle:

Principle #3Local quality

2Manufacturing precision

If image warping is performed on the entire image to provide parallax, then the 3D effect is improved, but the processing time and computational complexity increase

Engineering Contradiction:

Improvequality of 3D effectVSAvoidprocessing time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The image is divided into foreground and background areas, and image warping is performed separately on each segment. This segmentation allows parallel processing and reduces the overall processing time while maintaining the quality of the 3D effect through targeted warping operations on specific regions.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Image warping is applied selectively to specific regions (foreground and background areas) rather than the entire image. This partial application of the warping operation reduces computational complexity and processing time while still achieving the desired 3D parallax effect in the critical regions.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11037362B2Method and apparatus for generating 3D virtual viewpoint image

Publication Date: 2021.06.15 ELECTRONICS & TELECOMM RES INST

AI summary

A method and an apparatus for generating a three-dimension (3D) virtual viewpoint image including: segmenting a first image into a plurality of images indicating different layers based on depth information of the first image at a gaze point of a user; and inpainting an area occluded by foreground in the plurality of images based on depth information of a reference viewpoint image are provided.