3D Panoramic Image Stitching With Depth-Aware Pixel Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for forming panoramic images in real-time are inaccurate due to parallax errors caused by assuming the three-dimensional position of objects, which leads to visual quality issues in automotive applications like ADAS and parking assistance systems.
Innovation Solution
A method that integrates depth sensor data to reconstruct a three-dimensional map of the environment, combines this with a pre-calculated lookup table (LUT) to accurately determine pixel positions and weights, and linearly combines image pixels for panoramic image formation using synchronized and calibrated cameras.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional stitching procedure with homography estimation is used, then real-time panoramic image formation is achieved, but parallax errors occur due to incorrect assumption of object positions
Solution Approach 1:
The patent introduces depth sensors as an intermediary device to obtain accurate depth information about objects in the environment. This depth information acts as a mediator between the image data from multiple cameras and the stitching process, enabling accurate estimation of object positions without relying on incorrect assumptions. The depth data from sensors like LIDAR or time-of-flight cameras provides the missing spatial information needed to resolve parallax errors while maintaining real-time performance.
Solution Approach 2:
The patent changes the parameters used in the stitching process by incorporating depth information (z-coordinate) alongside the traditional two-dimensional image coordinates. Instead of assuming objects are at infinity or using only 2D homography, the system uses 3D parameters including depth to calculate accurate projection matrices and perform perspective-aware stitching. This parameter expansion from 2D to 3D space resolves the position accuracy issue while maintaining computational efficiency for real-time operation.
2Measurement precision
If depth sensor data is integrated to reconstruct three-dimensional map, then object position accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing lookup tables (LUTs) that contain pre-computed projection matrices and stitching parameters for various object positions and depths. During real-time operation, the system only needs to query these pre-computed tables using the current depth sensor data, rather than performing complex iterative optimization and homography estimation. This shifts the computational burden from real-time processing to an offline preprocessing stage, significantly reducing real-time computational complexity while maintaining high position accuracy.
3Manufacturing precision
If multiple synchronized cameras and depth sensors are used, then panoramic image quality is improved, but system complexity and synchronization requirements increase
Solution Approach 1:
The patent merges the data from multiple cameras and depth sensors into a unified coordinate system and processing pipeline. By integrating the depth information from sensors with the image data from cameras in a coordinated manner, the system achieves accurate 3D reconstruction and panoramic stitching. The synchronized acquisition and combined processing of multi-sensor data improve panoramic image quality by providing both visual and depth information, while the unified processing framework manages the system complexity through integrated calibration and coordinate transformation.
Data Source
AI summary
A method is described for forming a panoramic image (I); the method comprising: receiving (B1) a plurality of images (I1, I2 . . . Ir) of an environment from a corresponding plurality of cameras (C1, C2 . . . Cr) at a given time instant (t), receiving (B1) data on the position of the points relative to the physical objects of said environment shot by said plurality of cameras by at least one depth sensor (L1, L2 . . . Lf) at said given time instant, processing (B2) data obtained from the at least one depth sensor to construct the distance of all the objects contained inside said environment by a virtual camera (C), obtaining a three-dimensional map of the positions of the points relative to the physical objects of said environment, combining (B3) said three-dimensional map with the information contained in a previously made and stored lookup table (LUT) and wherein a cell of said lookup table (LUT) is allocated to each pixel of the plurality of pixels which compose said panoramic image observed by said virtual camera (C), said cell storing the information on the pixels (Pi) of said plurality of images corresponding to the position of the single point (P) of the three-dimensional map and information on the weights (Wi) associated with said corresponding pixels of said plurality of images, linearly combining (B4) said pixels of said plurality of images as a function of their weight obtaining said panoramic image.