Multimodal Ground Truth Sampling With Valid Pose Transformations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multimodal ground truth sampling methods fail to incorporate object-level transformations such as uniform scaling, translation, and rotation, limiting the diversity and accuracy of learned object detection models trained on multimodal datasets.
Innovation Solution
A method for multimodal ground truth sampling that applies valid pose transformations to source objects, creating transformed objects for synthetic image and LiDAR data, while rejecting unrealistic or occluded transformations, thereby enhancing dataset diversity and robustness without manual data collection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If current multimodal ground truth sampling methods are used, then the basic object detection task can be performed, but the diversity and accuracy of the trained model is limited due to lack of object-level transformations
Solution Approach 1:
The patent applies dynamic pose transformations (rotation, translation, scaling) to source objects to generate transformed objects with varying orientations and positions. This dynamic transformation process creates diverse training data that covers multiple object poses and viewpoints, thereby improving both dataset diversity and model accuracy simultaneously
Solution Approach 2:
The patent creates synthetic training data by copying and transforming existing source objects. Source objects are extracted from original images and point clouds, then replicated multiple times with different pose transformations to generate augmented training samples. This copying approach expands dataset diversity while maintaining the underlying object characteristics for accurate model training
2Adaptability or versatility
If pose transformations are applied to source objects, then dataset diversity is improved, but unrealistic or occluded transformations may be generated
Solution Approach 1:
The patent implements a feedback mechanism through rejection sampling where generated transformed objects are evaluated against realism criteria. Transformations that produce unrealistic appearances or occlusions are rejected and regenerating new transformations. This feedback loop ensures only high-quality, realistic transformations are included in the final training dataset
Solution Approach 2:
The patent extracts and removes unrealistic or occluded transformations from the set of generated pose transformations. By filtering out invalid transformations through rejection sampling, the method isolates only the realistic transformations suitable for training, thereby maintaining manufacturing precision while preserving diversity
3Productivity
If synthetic data is generated through ground truth sampling, then manual data collection is avoided, but the quality of synthetic data may be insufficient
Solution Approach 1:
The patent uses copying by extracting source objects from original data and creating synthetic copies through pose transformations. This copying mechanism enables efficient automated data generation without manual collection, while the quality is maintained through careful transformation parameters and rejection sampling that filters out low-quality synthetic samples
Solution Approach 2:
The patent applies parameter changes by varying pose parameters (rotation angles, translation vectors, scaling factors) to generate diverse synthetic objects. These parameter transformations allow efficient generation of multiple variants from single source objects, improving productivity while the quality is controlled through realistic parameter ranges and validation criteria
Data Source
AI summary
In some embodiments, a method of multimodal ground truth sampling for creating synthetic multimodal training data is provided, the method performed by one or more processors, the method comprising: selecting a source object from a dataset; determining a valid pose transformation from a set of proposed pose transformations; applying the valid pose transformation to the source object to create a transformed object; generating synthetic image data based on the transformed object and a destination image; generating synthetic point cloud data based on the transformed object and a destination point cloud; and training a computer vision machine learning model from synthetic multimodal training data comprising the synthetic image data and the synthetic point cloud data.