Free-Viewpoint Video Generation Using 3D Model Reference Areas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods face challenges in generating high-resolution free-viewpoint videos from multi-viewpoint videos of various viewpoints due to difficulties in specifying corresponding areas and accurately estimating pixel components.
Innovation Solution
A method that uses a three-dimensional model to specify reference areas in multiple videos and estimates pixel components with higher resolution, potentially through upsampling and parallelization, to generate a free-viewpoint video with a virtual viewpoint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a single high resolution image is generated from multiple low resolution input images, then the resolution is improved, but the accuracy of specifying corresponding areas and estimating pixel components deteriorates
Solution Approach 1:
The patent divides the image processing task into multiple stages: first generating a low-resolution free-viewpoint image from multi-viewpoint images, then segmenting this into target areas, and finally generating high-resolution images only for those target areas. This segmentation allows the system to maintain accuracy in corresponding area specification while achieving high resolution where needed.
Solution Approach 2:
The patent performs preliminary processing by first generating a low-resolution free-viewpoint image and creating a three-dimensional model before attempting high-resolution generation. This preliminary action establishes the spatial relationships and corresponding areas between different viewpoints, which then guides the subsequent high-resolution pixel component calculation, resolving the contradiction between resolution and accuracy.
2Manufacturing precision
If pixel components are calculated with high resolution, then the quality of free-viewpoint video is improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies local quality by calculating pixel components at high resolution only for specific target areas identified in the low-resolution free-viewpoint image, rather than processing the entire image at high resolution. This selective approach maintains high quality where needed while reducing overall computational complexity.
Solution Approach 2:
The patent segments the image processing into multiple stages where the computational burden is distributed: low-resolution processing for overall structure and target area identification, followed by high-resolution processing only for selected target areas. This segmentation reduces the total computational complexity compared to generating the entire high-resolution free-viewpoint image directly.
3Measurement precision
If a three-dimensional model is generated from multiple videos, then the accuracy of specifying reference areas is improved, but the processing time and computational resources increase
Solution Approach 1:
The patent performs the three-dimensional model generation as a preliminary action that is reused across multiple processing steps. By establishing the three-dimensional model and corresponding area relationships early, the system avoids redundant computations in subsequent high-resolution pixel component calculations, thereby reducing overall processing time while maintaining high accuracy.
Solution Approach 2:
The three-dimensional model serves multiple functions: it specifies reference areas, establishes spatial relationships between viewpoints, and guides high-resolution pixel component calculation. This multi-functionality reduces the need for separate processing steps, thereby reducing overall processing time while maintaining accuracy in reference area specification.
Data Source
AI summary
A free-viewpoint video generating method includes obtaining a multi-viewpoint video including N videos of N viewpoints (N being an integer greater than 1), and generating a free-viewpoint video of a virtual viewpoint based on the multi-viewpoint video such that the free-viewpoint video shows target areas and has a first resolution higher than each of resolutions of the N videos, the virtual viewpoint being different from the N viewpoints. For each target area, the generating includes: referring to a three-dimensional model generated based on the N videos to specify N reference areas which are shown by the N videos and which correspond to the each target area in the free-viewpoint video; and calculating components of pixels to represent the each target area in the free-viewpoint video, based on components of pixels to represent the N reference areas in the N videos.