3D Occupancy Prediction Using Multi-Scale Voxel Query Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D occupancy prediction technologies face challenges such as information loss and high computational complexity when converting 2D image data to 3D, leading to low prediction accuracy and difficulty in real-time processing, especially in autonomous vehicles and drones.
Innovation Solution
A method involving multi-scale image feature extraction, local cluster feature vector generation, 3D voxel query mapping through attention operations, and decoding with voxel upsampling and consistency regularization to predict 3D occupancy states and semantic classes, using 2D image segmentation supervised learning and contrastive learning techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If typical techniques convert 2D image data to 3D space, then 3D occupancy prediction can be performed, but information loss occurs and prediction accuracy decreases
Solution Approach 1:
The patent segments the 2D image into multiple patches and extracts features at multiple scales (low-level and high-level features). This segmentation allows preserving detailed spatial information while capturing semantic context, thereby reducing information loss during 2D to 3D conversion and improving 3D occupancy prediction accuracy.
Solution Approach 2:
The patent introduces multi-scale feature extraction that operates in both spatial and feature hierarchy dimensions. By combining low-level detailed features with high-level semantic features from multiple scales, the method creates a richer feature representation that reduces information loss when projecting to 3D space.
2Measurement precision
If high-resolution queries are used in typical techniques, then prediction accuracy may improve, but computational complexity increases making real-time processing difficult
Solution Approach 1:
The patent divides the query into multiple patches and processes them in parallel through the transformer encoder. This segmentation allows the system to maintain high-resolution queries while distributing computational load across multiple processing units, enabling real-time processing.
Solution Approach 2:
The patent uses multi-scale feature extraction where not all regions of the image require full processing depth. The system selectively applies different levels of feature extraction based on image regions and query requirements, reducing overall computational complexity while maintaining sufficient accuracy for real-time applications.
3Device complexity
If typical techniques only use low-level features of 2D images, then processing is simpler, but prediction accuracy decreases
Solution Approach 1:
The patent segments feature extraction into multiple scales, processing low-level detailed features and high-level semantic features separately through different transformer encoder depths. This allows the system to combine simple low-level processing with sophisticated high-level processing, achieving high accuracy without excessive overall complexity.
Solution Approach 2:
The patent applies different processing qualities to different parts of the feature extraction. Low-level features receive detailed processing for spatial accuracy, while high-level features receive semantic processing for object understanding. This local differentiation of processing quality enables high prediction accuracy with manageable overall complexity.
Data Source
AI summary
A processor-implemented method with three-dimensional (3D) occupancy prediction learning includes extracting multi-scale image feature vectors from received two-dimensional (2D) image data, generating a local cluster feature vector by clustering the extracted multi-scale image feature vectors, mapping the local cluster feature vector to a 3D space through an attention operation using a learnable voxel query; decoding a 3D voxel query generated according to the mapping result, and predicting a 3D occupancy state and a semantic class for a space, based on the decoding result.