Image Processing Apparatus for Accurate Virtual Viewpoint Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating virtual viewpoint images around a reference point, such as a vehicle, suffer from distortion and tilting due to the use of predetermined upright plane shapes that do not accurately represent the original object shapes, leading to inaccurate image mapping and poor image quality.
Innovation Solution
An image processing apparatus that integrates first three-dimensional shape data of objects near the reference point with second three-dimensional shape data of the surrounding environment, formed from flat or curved planes, to create accurate texture-mapped three-dimensional shape data, allowing for precise virtual viewpoint image generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a predetermined upright plane shape is used for three-dimensional shape data, then the device complexity is reduced and ease of manufacture is improved, but the manufacturing precision and reliability of the virtual viewpoint image deteriorate due to shape mismatch with actual objects
Solution Approach 1:
The patent divides the three-dimensional space into multiple regions (first region for near objects, second region for distant objects, third region for far objects) and assigns different plane shape generation methods to each region. This segmentation allows the system to use simple predetermined shapes for distant regions while applying accurate object-shaped planes for near regions, thereby resolving the contradiction between ease of manufacture and manufacturing precision.
Solution Approach 2:
The patent applies different qualities of plane shapes to different spatial regions: predetermined upright plane shapes are used for distant regions where high precision is less critical, while object-specific three-dimensional shape data is used for near regions where accuracy is paramount. This local differentiation optimizes both computational efficiency and image quality according to spatial requirements.
2Device complexity
If a predetermined upright plane shape is used, then the device complexity is reduced, but the distortion and tilting in the virtual viewpoint image increase
Solution Approach 1:
The patent segments the mapping process into multiple stages corresponding to different spatial regions. By separating the handling of near objects (using accurate object-shaped planes) from distant objects (using simpler predetermined planes), the system reduces overall complexity while maintaining reliability where it matters most.
Solution Approach 2:
Instead of using a single uniform plane shape generation method for all objects, the patent inverts the conventional approach by using different methods for different regions. The system prioritizes accuracy for near objects and accepts simplification for distant objects, effectively inverting the traditional uniform treatment approach.
3Manufacturing precision
If object-specific three-dimensional shape data is obtained for all objects, then the manufacturing precision and image accuracy are improved, but the loss of time and computational resources increase
Solution Approach 1:
The patent segments the processing workload by spatial region, applying computationally intensive object-specific shape generation only to near objects in the first region, while using pre-defined shapes for distant objects. This segmentation significantly reduces total processing time while maintaining precision where it is most visible and important.
Solution Approach 2:
The patent applies the principle of partial action by generating accurate object-specific three-dimensional shape data only for near objects where it is most needed, rather than for all objects. This partial application of the intensive processing method optimizes the balance between precision and time consumption.
Data Source
AI summary
First, an image processing apparatus obtains data of a captured image obtained by image capturing with an image capturing apparatus that captures an image of a surrounding of a reference point, and obtains distance information indicating a distance from the reference point to an object present in a vicinity of the reference point. Next, the image processing apparatus obtains first three-dimensional shape data corresponding to a shape of the object, based on the distance information. Then, the image processing apparatus obtains second three-dimensional shape data that corresponds to the surrounding of the reference point other than the object and that is formed of one or more flat planes or curved planes. Then, the image processing apparatus obtains third three-dimensional shape data in which the first three-dimensional shape data and the second three-dimensional shape data are integrated, and maps the captured image to the third three-dimensional shape data.