3D Anchor Points for Temporal Consistency in Computer Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for computer vision tasks face inaccuracies when dealing with images that depict small portions of surfaces or objects with few features, as they lack sufficient object distinguishing information, leading to reduced accuracy and efficiency.
Innovation Solution
The implementation of a machine learning model that utilizes 3D anchor points storing prior information about a physical setting, allowing the model to reason about future images with temporal and geometric consistency by associating semantic labels and normal directions with 3D locations, thereby providing context-aware determinations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models process individual images with limited features, then processing speed is maintained, but accuracy deteriorates for images with few object distinguishing features
Solution Approach 1:
The system performs preliminary action by pre-processing images to extract and store 3D anchor points with semantic labels and geometric features before they are needed for labeling. This allows the model to access pre-computed object distinguishing information from multiple views and distances, improving accuracy for images with limited features without sacrificing processing speed during inference.
Solution Approach 2:
The system transitions from 2D image space to 3D world space by creating 3D anchor points that store semantic information. This dimensional transformation allows the model to leverage geometric consistency across multiple images and viewpoints, providing additional object distinguishing information that is not available in single 2D images, thereby resolving the contradiction between processing speed and labeling accuracy.
2Stability of the object's composition
If 3D anchor points are stored and used for future images, then temporal and geometric consistency is improved, but device complexity increases
Solution Approach 1:
The system segments the complex task of maintaining temporal and geometric consistency into manageable components: extracting 3D anchor points from images, storing them in a structured format with semantic labels, and efficiently querying them during inference. This segmentation reduces system complexity by breaking down the overall complexity into modular, independent operations that can be implemented and maintained separately.
3Productivity
If machine learning models use only current image information, then processing efficiency is maintained, but accuracy deteriorates for close-up images with limited context
Solution Approach 1:
The system creates copies of semantic information from 3D anchor points that were derived from images taken at different distances and angles. These copied semantic labels are then used to augment the current image processing, providing additional object distinguishing information for close-up images without requiring re-capture of reference images, thus maintaining processing efficiency while improving labeling accuracy.
Data Source
AI summary
A machine learning model is trained and used to perform a computer vision task such as semantic segmentation or normal direction prediction. The model uses a current image of a physical setting and input generated from three dimensional (3D) anchor points that store information determined from prior assessments of the physical setting. The 3D anchor points store previously-determined computer vision task information for the physical setting for particular 3D points locations in a 3D worlds space, e.g., an x, y, z coordinate system that is independent of image capture device pose. For example, 3D anchor points may store previously-determined semantic labels or normal directions for 3D points identified by simultaneous localization and mapping (SLAM) processes. The 3D anchor points are stored and used to generate input for the machine model as the model continues to reason about future images of the physical setting.