Deep Learning Panoramic Stitching for Real-Time Parallax Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image matching methods, including deep learning-based approaches, struggle with generating panoramic images with wide viewing angles due to high computational complexity, reliance on feature matching, and the need for special hardware, which limits real-time processing and inclusion of dynamic objects.
Innovation Solution
A method and apparatus using a deep learning network to generate panoramic images by configuring a data set, extracting feature maps, estimating homographies and blending maps, and deriving flow maps from multiple camera inputs, without requiring additional calculations or separate hardware, and utilizing weakly supervised learning to correct and blend images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional image matching methods use feature matching and energy function optimization, then image matching can be performed, but computational complexity greatly increases making real-time processing difficult
Solution Approach 1:
The patent replaces conventional mechanical/image processing methods (feature matching, homography estimation, energy function optimization) with a deep learning-based neural network system. The neural network directly processes input images to generate panoramic output images, substituting the complex multi-step conventional pipeline with a learned end-to-end mapping that achieves real-time processing while maintaining matching accuracy.
2Measurement precision
If conventional methods perform local homography estimation and energy function optimization for each image, then 2D transform selection can be achieved, but computational complexity increases making high-resolution panoramic image matching difficult in real time
Solution Approach 1:
The patent performs preliminary action by pre-training the neural network on large datasets of images with known transformations. This pre-learning enables the network to directly predict panoramic transformations during inference without performing computationally expensive homography estimation or energy function optimization at runtime, achieving both high precision and real-time processing speed.
3Reliability
If hardware fixing method is used to acquire answer image, then parallax-free image can be obtained, but special hardware is required and dynamic objects cannot be included
Solution Approach 1:
The patent replaces the hardware-based parallax correction method with a software-based deep learning approach. The neural network learns to correct parallax effects by training on images captured from multiple cameras, enabling the system to achieve parallax-free panoramic images using only standard multi-camera setups without specialized hardware fixing mechanisms.
4Ease of manufacture
If unsupervised learning method is used, then data can be generated by cropping, but panoramic image with wide viewing angle cannot be obtained
Solution Approach 1:
The patent merges multiple cropped images from different cameras and perspectives into a single panoramic image with wide viewing angle (180-360 degrees). The neural network integrates information from multiple input images, combining their respective fields of view to produce a comprehensive panoramic output that exceeds the viewing angle of any individual camera while maintaining the simplicity of data generation from standard camera feeds.
Data Source
AI summary
Disclosed herein are a method and apparatus for generating a panoramic image. The method includes configuring a data set for training a deep learning network based on K images, extracting an encoding feature map, a skip connection feature map, and a decoding feature map based on K/2 images, among the images, estimating multiple homographies based on the encoding feature map, estimating a flow adjustment map, a preprocessing blending map, a weight map, and a post-processing blending map based on the decoding feature map, deriving a flow map based on the multiple homographies and the flow adjustment map, generating a preprocessed image based on the preprocessing blending map, generating multiple warped images based on the preprocessed image and the flow map, generating a matching image based on the warped images and the weight map, and correcting the matching image based on the matching image and the post-processing blending map.