Triangular Warping for Virtual Viewpoint Image Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image-warping technologies for virtual viewpoint synthesis in VR suffer from artifacts and degraded quality, particularly when combining rotation and motion with large overlap and viewpoint changes, leading to distorted depth images and poor synthesis quality.
Innovation Solution
An apparatus and method that utilize triangular warping and blending, where multiple viewpoint images are back-projected onto a 3D world coordinate system, and then projected onto a virtual viewpoint image coordinate system, using sinusoidal projection to calibrate triangle quality values and calculate blending weights based on depth values, to eliminate artifacts and improve image synthesis quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If median filter is used to eliminate holes before backward warping, then hole filling is achieved, but depth image distortion and artifacts occur in foreground object areas
Solution Approach 1:
The patent applies different processing strategies to different regions of the depth image. Foreground regions use depth values from the current viewpoint image, while background regions use depth values from the warped depth image. This local differentiation prevents foreground distortion while still filling background holes effectively.
Solution Approach 2:
The patent performs forward warping to generate a warped depth image before backward warping. This preliminary action creates a reference depth map that can be used to fill holes in the final backward-warped depth image, preventing the need for median filtering that would distort foreground objects.
2Adaptability or versatility
If image warping is performed with large overlap and viewpoint changes, then comprehensive spatial information is utilized, but synthesis quality deteriorates due to artifacts
Solution Approach 1:
The patent transitions from 2D image warping to 3D coordinate-based warping. By back-projecting images to a 3D world coordinate system and then projecting to the virtual viewpoint, the method maintains geometric accuracy even with large viewpoint changes, eliminating artifacts while utilizing comprehensive spatial information.
Solution Approach 2:
The patent introduces a 3D world coordinate system as an intermediary between the input viewpoint images and the virtual viewpoint image. This intermediary representation allows for accurate geometric transformation and blending, preventing artifacts that would otherwise occur with direct 2D warping.
3Device complexity
If conventional warping methods are used, then processing simplicity is maintained, but synthesis quality and artifact elimination are insufficient
Solution Approach 1:
The patent divides the warping process into two distinct segments: forward warping to generate a warped depth image, and backward warping to generate the final virtual viewpoint image. This segmentation allows each step to be optimized independently, improving overall quality while maintaining reasonable complexity.
Data Source
AI summary
Disclosed herein are an apparatus and method for synthesizing virtual viewpoint image. The apparatus for synthesizing virtual viewpoint images includes one or more processors, and an execution memory for storing at least one program that is executed by the one or more processors, wherein the at least one program is configured to receive multiple viewpoint images from cameras that capture images of multiple viewpoints, and back-project the multiple viewpoint images onto a world coordinate system in a three-dimensional (3D) space, generate warped images by performing triangular warping of projecting the multiple viewpoint images, back-projected onto the world coordinate system, onto a virtual viewpoint image coordinate system, and generate a finally synthesized virtual viewpoint image by blending the warped images.