Multi-Resolution Image Processing for Quantized 3D Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for 3D perception in automated driving using single cameras face challenges with memory intensity and overflow issues when quantizing models, particularly due to the use of integral images for feature aggregation over large voxel grids, leading to artifacts and incorrect results.
Innovation Solution
The method involves extracting feature maps at different resolutions from a vertical image plane and transforming them into a horizontal image plane, allowing for pixel access adjustments based on distance from the camera, thereby reformulating integral image computation into image resizing and pixel access, which are quantizable operations, facilitating efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If integral images are used for feature aggregation over large voxel grids, then feature extraction completeness is improved, but memory consumption increases and overflow issues occur during quantization
Solution Approach 1:
The patent segments the feature aggregation process by replacing integral image computation with direct pixel access from pre-extracted feature maps at multiple resolutions. Instead of computing cumulative sums over large voxel grids, the system divides the problem into discrete pixel accesses at different resolution levels, significantly reducing memory requirements while maintaining feature extraction completeness.
Solution Approach 2:
The patent transitions from 3D voxel grid aggregation to 2D feature map pixel access by projecting 3D spatial information into 2D feature maps at multiple resolutions. This dimensional reduction allows efficient feature aggregation without the memory overhead of maintaining large 3D integral images, solving the contradiction between feature completeness and memory consumption.
2Measurement precision
If integral images are used for feature aggregation, then feature extraction accuracy is improved, but quantization causes overflow and artifacts
Solution Approach 1:
The patent replaces the expensive and problematic integral image computation with simpler, quantization-friendly pixel access operations. Instead of maintaining large integral images that are prone to overflow during quantization, the system uses direct pixel accesses from feature maps that can be efficiently quantized to int8 or int16, ensuring reliability while maintaining accuracy.
Solution Approach 2:
The patent changes the computational parameters from integral image cumulative sums to direct pixel value accesses at multiple resolutions. This parameter change makes the operations compatible with quantized arithmetic, eliminating overflow issues while preserving feature extraction accuracy through multi-resolution feature map access.
3Measurement precision
If high resolution feature maps are used for distant objects, then detection accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent applies local quality by using different feature map resolutions for different spatial regions. Distant objects in the bird's-eye view access high-resolution feature maps for accurate detection, while closer objects use lower-resolution maps. This selective resolution approach maintains detection accuracy for critical distant objects while reducing overall processing complexity.
Solution Approach 2:
The patent implements dynamic resolution selection where the feature map resolution used for each pixel in the bird's-eye view is determined by its distance from the camera. This dynamic approach allows the system to adaptively use high resolution only where needed for distant objects, balancing detection accuracy with processing efficiency.
4Area of stationary object
If voxel grid size is increased to cover larger areas, then scene coverage is improved, but memory intensity increases
Solution Approach 1:
The patent reduces the memory intensity of large scene coverage by projecting 3D voxel grid information into 2D feature maps at multiple resolutions. This dimensional transformation allows the system to cover large scenic areas without proportionally increasing memory usage, as the 2D feature maps require significantly less memory than equivalent 3D voxel grids.
Solution Approach 2:
The patent segments the large scene coverage problem into multiple resolution levels, where different portions of the scene are represented at appropriate resolutions. This segmentation allows comprehensive scene coverage while managing memory intensity by using lower resolutions for areas requiring less detail.
Data Source
AI summary
A method for processing image data captured by a camera in a vertical image plane includes extracting a feature map from the image data captured by the camera in the vertical image plane by a feature extractor, adjusting a resolution of the extracted feature map depending on a position of a portion of the image data in a horizontal image plane relative to the camera to obtain a feature map with adjusted resolution, and transforming the portion of the image data from the vertical image plane to the horizontal image plane based on the feature map with the adjusted resolution.