Neural Network Training Data Selection via ODD Bucketing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Selecting training data for neural networks is resource-intensive, especially when dealing with large datasets, and existing methods lack efficiency in approximating target distributions for diverse training scenarios.
Innovation Solution
A system that automatically selects and curates scenes by partitioning unlabelled data into buckets based on Operational Design Domain (ODD) values, using algorithms to ensure diversity and optimize the selection process to match a target distribution, thereby reducing the computational burden and improving data efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods are used to select training data from large datasets, then comprehensive coverage of data distribution is achieved, but computational resources and time consumption increase significantly
Solution Approach 1:
The patent segments the large unlabelled dataset into multiple buckets based on Operational Design Domain (ODD) values. Each bucket represents a specific subset of the data space characterized by particular ODD properties. This segmentation enables parallel processing and reduces the computational burden of evaluating the entire dataset sequentially, while still maintaining comprehensive coverage of the target distribution through strategic sampling from each segment.
Solution Approach 2:
The system performs preliminary actions by pre-computing ODD values and organizing data into buckets before the actual training data selection process. This preliminary organization creates an efficient data structure that allows rapid querying and sampling during training data selection, avoiding the need to re-process the entire dataset each time training data needs to be selected.
2Measurement precision
If more computational resources are allocated to data selection, then better approximation of target distribution is achieved, but resource efficiency decreases
Solution Approach 1:
The patent implements a self-service mechanism where the system automatically selects training data by computing ODD values and performing sampling based on target distribution requirements, without requiring manual intervention or extensive human-curated datasets. The algorithm autonomously identifies representative samples from each ODD bucket, reducing the need for expensive manual labeling and curation processes while maintaining high accuracy in approximating the target distribution.
Solution Approach 2:
Instead of processing the entire large dataset to ensure complete coverage, the system applies partial action by selectively sampling from representative buckets. This approach achieves sufficient approximation of the target distribution by focusing computational effort on key segments rather than exhaustively processing all data points, thereby reducing overall resource consumption while maintaining acceptable accuracy levels.
3Adaptability or versatility
If manual curation methods are used to ensure data diversity, then data quality is improved, but labor requirements and costs increase
Solution Approach 1:
The patent replaces manual mechanical curation processes with an automated computational system that uses ODD value computation and algorithmic sampling to ensure data diversity. Instead of relying on human experts to manually select diverse training samples, the system automatically calculates ODD values for data points and uses these values to guide sampling from different buckets, thereby maintaining data diversity while eliminating the need for manual labor and reducing curation process complexity.
Data Source
AI summary
Apparatuses, systems, and techniques to automatically select training data. In at least one embodiment, training data is automatically selected based on, for example, metadata associated with the training data.