Virtual View Generation via Epipolar Reprojection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for generating virtual views from physical camera images often result in undesired artifacts when merging data from different sources, failing to preserve the three-dimensional structure and allowing seamless perspective changes.
Innovation Solution
A method and system that utilize a capturing device with a physical camera and a depth sensor to determine the actual and desired poses of the camera, define epipolar geometry, and resample depth information to generate a virtual view, minimizing a direction cost function to identify target pixels and create a disparity map for perspective change.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If image data from multiple physical cameras at different positions are merged to generate virtual views, then the ability to change perspective is improved, but undesired artifacts are introduced and three-dimensional structure is not preserved
Solution Approach 1:
The patent introduces a virtual camera model as an intermediary between physical cameras and the desired virtual view. This virtual camera serves as a mediator that defines the epipolar geometry and projection relationships, allowing seamless perspective changes without directly merging multiple physical camera images, thus avoiding artifacts while preserving 3D structure.
Solution Approach 2:
The patent changes the fundamental parameters from merging multiple image sources to using a single physical camera with depth information. By transforming the problem into a single-camera setup with epipolar geometry constraints and depth-based projection, the system achieves high-quality virtual views without the artifacts caused by multi-camera merging.
2Adaptability or versatility
If multiple images from different perspectives are used to generate virtual views, then perspective flexibility is improved, but merging artifacts are introduced
Solution Approach 1:
The patent extracts the essential information (depth data and epipolar geometry) from multiple perspective images and consolidates it into a unified virtual camera model. This extraction approach eliminates the need to merge multiple image sources directly, thereby removing the source of merging artifacts while maintaining virtual view flexibility.
3Manufacturing precision
If depth information is resampled in epipolar coordinates to generate virtual views, then three-dimensional structure preservation is improved, but computational complexity increases
Solution Approach 1:
The patent transforms the depth information from the original camera coordinate system to the virtual camera's epipolar coordinate system. This dimensional transformation allows depth data to be resampled along epipolar lines, preserving three-dimensional structure while managing computational complexity through structured processing of depth maps and projection matrices.
Data Source
AI summary
Systems and methods for generating a virtual view of a virtual camera based on an input scene are described. A capturing device typically includes a physical camera and a depth sensor and captures an input scene. A controller determines an actual pose of the capturing device and a desired pose of the virtual camera for showing the virtual view. The controller defines an epipolar geometry between the actual pose of the capturing device and the desired pose of the virtual camera. The controller generates an output image for the virtual camera based on an epipolar relation between the actual pose of the capturing device, the input scene, and the desired pose of the virtual camera.