Surround View Image Blending via Precomputed Seamline Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing surround view systems in multi-camera environments, especially in automotive applications, face inefficiencies and inaccuracies due to the reliance on GPUs for real-time pixel remapping, which is costly and can produce artifacts from inaccurate stitching, especially when handling arbitrary virtual viewpoints.
Innovation Solution
A configuration process generates a three-dimensional surface representation and defines seamlines, calculating blend weights for pixels to populate a blend table, which is then partitioned based on a cost function, allowing for accelerated rendering by skipping blending steps for pixels not on seamlines, and utilizing ASICs or SoCs for real-time processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If GPU-based solutions are used to handle pixel remapping for arbitrary virtual viewpoints, then rendering accuracy and flexibility are improved, but system cost and overhead increase significantly
Solution Approach 1:
The image processing task is segmented into two distinct phases: an offline configuration phase that generates pixel remapping data and blend tables, and an online rendering phase that uses pre-computed data for real-time image synthesis. This segmentation allows complex processing to be done offline, reducing real-time computational requirements and system cost.
Solution Approach 2:
Pixel remapping configurations and blend tables are pre-computed during an offline configuration phase for various virtual viewpoints. This preliminary action stores the results in lookup tables, enabling fast real-time rendering without requiring expensive GPU-based pixel remapping during actual operation.
2Adaptability or versatility
If GPU-based solutions are used for real-time pixel remapping, then arbitrary virtual viewpoints are supported, but processing overhead increases
Solution Approach 1:
Pixel remapping data and blend tables for multiple virtual viewpoints are pre-computed and stored during an offline configuration phase. During real-time operation, the system simply retrieves pre-computed data from lookup tables, eliminating the need for time-consuming GPU-based pixel remapping while maintaining support for arbitrary virtual viewpoints.
Solution Approach 2:
The system creates copyable lookup tables containing pixel remapping configurations and blend weights for various virtual viewpoints. These tables can be stored and quickly referenced during rendering, allowing the system to support multiple viewpoints without repeating the expensive pixel remapping computation for each viewpoint change.
3Productivity
If predetermined remapping configurations are used for limited vantage points, then processing speed is improved, but stitching accuracy deteriorates due to artifacts
Solution Approach 1:
The system performs preliminary computation of accurate pixel remapping configurations and blend tables for multiple predetermined vantage points during an offline phase. This allows the online rendering process to use pre-computed accurate data for fast rendering while maintaining high stitching quality, avoiding the artifacts that would result from using simplified or approximate remapping methods.
Solution Approach 2:
The system pre-computes different pixel remapping configurations and blend tables for multiple virtual viewpoints with different parameters (position, orientation, field of view). This allows the rendering system to quickly switch between viewpoints by selecting the appropriate pre-computed configuration, maintaining both speed and accuracy for each specific viewpoint.
4Stability of the object's composition
If blending is performed for all pixels, then seamline continuity is improved, but processing time increases
Solution Approach 1:
The system applies blending selectively only to pixels located near seamlines, while using non-blended pixels from the pre-computed lookup tables for the majority of the image. The blend tables store weights that indicate which pixels require blending and to what extent, enabling the system to maintain seamline continuity while minimizing processing time by avoiding unnecessary blending operations.
Solution Approach 2:
The image is effectively segmented into regions requiring blending (near seamlines) and regions not requiring blending (away from seamlines). The blend tables identify and separate these regions, allowing the rendering system to apply blending only where necessary to maintain continuity, while using faster non-blended pixel retrieval for the majority of the image area.
Data Source
AI summary
An apparatus comprising a memory and one or more processing circuits is provided. The memory stores a blend table having blend weights. The processing circuits, for partitions of the blend table: determine whether a subset of the pixels associated with the partition includes pixels associated with seamlines defined in a three-dimensional surface representation of the scene. If none of the subset of the pixels are associated with the seamlines, the processing circuits populate a region of the virtual image corresponding to the partition with pixel values from an image captured by one of the plurality of image capture devices. If one or more of the subsets of the pixels is associated with the seamlines, the processing circuits populate the region of the virtual image associated with the partition with blended pixel values from two or more images captured by two or more of the plurality of image capture devices.