Image Feature Map Resizing for Memory-Efficient 3D Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for 3D perception in automated driving using single cameras face challenges with memory intensity due to large voxel grids, particularly when quantized, leading to overflow issues and artifacts in integral image computations.
Innovation Solution
The method involves transforming image data from a vertical image plane to a horizontal image plane by adjusting feature map resolutions based on distance from the camera, using a Convolutional Neural Network (CNN) to extract and convert feature maps, allowing for pixel access instead of integral image computation, thereby addressing memory and overflow issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If integral image computation is used for orthographic feature transformation, then transformation accuracy is improved, but memory consumption increases and overflow issues occur when quantized
Solution Approach 1:
The patent segments the feature map processing into multiple resolutions. Instead of computing integral images for the entire high-resolution feature map, the system divides it into low-resolution and high-resolution parts, computing integral images only where necessary (for distant objects) and using direct pixel access for nearby objects. This segmentation reduces overall memory consumption while maintaining transformation accuracy where needed.
Solution Approach 2:
The patent applies local quality by using different computation strategies for different regions of the feature map. For regions corresponding to distant objects, integral image computation is used to maintain accuracy. For regions corresponding to nearby objects, direct pixel access is sufficient. This localized approach optimizes the balance between accuracy and memory usage.
2Measurement precision
If high-resolution feature maps are used for all image regions, then transformation accuracy is improved, but processing time and memory consumption increase
Solution Approach 1:
The patent implements dynamic resolution selection based on the position in the bird's-eye view image. The system dynamically chooses between low-resolution and high-resolution feature maps for different spatial regions, using low-resolution for distant regions and high-resolution for nearby regions. This dynamic approach reduces processing time and memory consumption while maintaining accuracy where it matters most.
Solution Approach 2:
The patent applies partial action by using high-resolution feature maps only for regions where it is necessary (nearby objects), rather than applying it uniformly across the entire image. For distant objects, low-resolution feature maps are sufficient, reducing the overall computational burden and processing time.
3Quantity of substance
If quantized models are used to reduce memory consumption, then memory efficiency is improved, but overflow issues and artifacts occur in integral image computations
Solution Approach 1:
The patent extracts and removes the problematic integral image computation from the quantized model pipeline. By separating the computation into regions where integral images are needed and regions where they are not, the system eliminates overflow issues in quantized regions while maintaining accuracy where integral images are computed in higher precision.
Solution Approach 2:
The patent introduces an intermediary mechanism that selectively applies integral image computation only where necessary. This intermediary layer decides which regions require integral image processing and which can use direct pixel access, thereby preventing overflow issues in quantized models while maintaining computation reliability where needed.
Data Source
Figure 1~3
AI summary
Provided is a method for processing image data captured by a camera in a vertical image plane. The method comprises extracting a feature map from the image data captured by the camera in one of the vertical image planes by a feature extractor, adjusting a resolution of the extracted feature map depending on a position of a portion of the image data in a horizontal image plane relative to the camera to obtain a feature map with adjusted resolution, and transforming the portion of the image data from the vertical image plane to the horizontal image plane based on the feature map with the adjusted resolution.