Neural Image Fusion With Overlap-Aware Feature Map Addition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for fusing image data from different acquisition ranges in ADAS/AD systems struggle to efficiently combine features from varying resolution levels, leading to challenges in object recognition, especially for objects partially contained in overlapping regions.
Innovation Solution
A method that involves receiving input image data from overlapping acquisition regions, determining feature maps for each image, computing output feature maps through convolution, and then fusing these maps element-by-element, considering the positional overlap to generate a unified fused feature map.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If image data from different acquisition ranges and resolution levels are processed separately by independent CNNs, then each image can be processed with appropriate resolution, but computing runtime increases significantly and weights must be reloaded for each image
Solution Approach 1:
The patent merges multiple images with different acquisition ranges and resolution levels into a single composite image by spatially aligning and concatenating them. This unified image is then processed by a single CNN, eliminating the need for multiple independent inferences and weight reloads, thus reducing computing runtime while maintaining object recognition accuracy
Solution Approach 2:
The patent segments the processing task by dividing the composite image into different acquisition range regions (first acquisition range, second acquisition range, and overlap region). Each region is processed with appropriate resolution levels, allowing the CNN to efficiently handle different spatial frequencies in different regions without requiring separate CNNs for each image
2Productivity
If image regions are concatenated like an art collage from different resolution levels, then processing can be unified in a single CNN, but objects in overlap regions cannot be easily identified as belonging to the same object
Solution Approach 1:
The patent introduces an overlap region as an intermediary zone between the first and second acquisition ranges. This overlap region contains corresponding portions from both acquisition ranges, allowing the CNN to receive redundant information about objects in the transition zone. The neural network can reliably identify objects spanning the boundary by analyzing the consistent features across the overlap region, thus maintaining object identification reliability while using a unified processing approach
3Measurement precision
If high resolution is used across the entire image, then all objects can be recognized in detail, but computing power requirements increase significantly
Solution Approach 1:
The patent applies local quality by assigning different resolution levels to different spatial regions of the image. The first acquisition range and second acquisition range are processed at their respective optimal resolutions, with the overlap region containing corresponding portions from both. This allows the system to maintain high detail recognition where needed while reducing computing power requirements in regions where lower resolution suffices
Data Source
AI summary
A method and system for fusing image data from an image acquisition sensor. The method includes: a) receiving input image data including: a first image which having a first region of a scene, and a second image which includes a second region of the scene, wherein the first and second regions overlap one another but are not identical; b) determining first and second feature maps using the first and second images, respectively; c) computing first and second output feature maps by convolutions of the first and second feature maps, respectively; d) computing a fused feature map through element-by-element addition of the first and second output feature maps, wherein the relative positions of the first and second regions are utilized, such that elements in the region of overlap are added. The method is runtime-efficient and fuses image data from one or more image acquisition sensors for an ADAS/AD vehicle system.