Scene Flow Estimation Using AI Motion Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing scene flow estimation methods struggle to accurately estimate movements between image frames, especially in complex scenes with varying textures and occlusions, which affects the precision of 3D structure and motion estimation.
Innovation Solution
A processor-implemented scene flow estimation method that inputs a frame pair into an AI network, obtaining motion embedding features and non-occluded-category label embedding features. These features are then fused to update the motion field, allowing for more accurate scene flow estimation by accounting for object categories and occlusions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional scene flow estimation methods are used, then the processing speed is faster, but the measurement precision of scene flow is lower
Solution Approach 1:
The patent segments the scene flow estimation task into multiple processing stages within the AI network: feature extraction from color and depth images, motion field initialization, iterative refinement with occlusion handling, and final scene flow computation. This segmentation allows each stage to specialize in specific aspects, improving overall accuracy while managing computational complexity through modular processing
Solution Approach 2:
The patent incorporates depth image information as an additional dimension beyond traditional color image analysis. By fusing 2D color features with 3D depth features, the system creates a multi-dimensional feature space that provides richer geometric context, significantly improving scene flow estimation accuracy in complex scenes with occlusions and varying textures
2Measurement precision
If AI network with feature fusion is used, then the measurement precision of scene flow is improved, but the loss of time in processing increases
Solution Approach 1:
The patent performs preliminary feature extraction from both color and depth images before the main scene flow computation. Motion fields are initialized using pre-extracted features, and occlusion maps are generated in advance. This preliminary processing organizes data structures and identifies key regions beforehand, reducing the computational burden during iterative refinement and overall processing time
Solution Approach 2:
The patent implements iterative refinement where the motion field is updated multiple times with progressively finer detail. In each iteration, only necessary regions are reprocessed based on occlusion detection and feature mismatch identification. This partial reprocessing approach balances accuracy improvement with computational efficiency, avoiding unnecessary full-scene reanalysis
3Reliability
If simple feature matching is used, then the device complexity is lower, but the reliability of scene flow estimation in complex scenes deteriorates
Solution Approach 1:
The patent fuses multiple feature types from different sources: color image features, depth image features, and motion field features. This composite feature representation combines the advantages of each modality - color provides texture information, depth provides geometric structure, and motion fields provide temporal context - creating a robust feature set that maintains reliability in complex scenes with occlusions and repetitive patterns
Solution Approach 2:
The patent introduces motion fields as an intermediary representation between feature extraction and final scene flow computation. Motion fields serve as a mediator that integrates information from color and depth features, providing a structured intermediate representation that captures temporal motion patterns. This intermediary layer simplifies the final scene flow estimation while maintaining reliability through multi-source information integration
Data Source
AI summary
A scene flow estimation method includes: inputting a frame pair into an artificial intelligence (AI) network, and obtaining therefrom a motion embedding feature and a non-occluded-category label embedding feature corresponding to a target pixel in the frame pair; and estimating a scene flow corresponding to the frame pair based on the motion embedding feature and the non-occluded-category label embedding feature, the frame pair includes a first frame and a second frame, the first frame including a first color image and a first depth image and the second frame including a second color image and a second depth image, the non-occluded-category label embedding feature includes category information of an object corresponding to a pixel pair in the frame pair, the pixel pair includes a first pixel of the first frame and a second pixel of the second frame, and the second pixel corresponds to the first pixel.