Training Dataset Curation Using Embedding Redundancy Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine learning models on large datasets requires significant computational resources and time due to redundant data, which can lead to longer training times and inefficient use of computing power.
Innovation Solution
A curation process that eliminates redundant training samples by identifying pairs with closely-matching embeddings using an approximate nearest neighbor search, setting a user-defined removal rate, and monitoring class and cross-class spread metrics to ensure diversity, resulting in a smaller, high-quality training dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large training dataset is used to train a machine learning model, then model accuracy is improved, but training time and computational resources increase significantly
Solution Approach 1:
The patent extracts and removes redundant training samples from the dataset by comparing embeddings of training samples and identifying pairs with closely-matching embeddings. By eliminating these redundant samples while preserving diverse and informative ones, the method reduces training time and computational resource requirements while maintaining model accuracy on par with or better than training on the full dataset.
2Reliability
If a large training dataset is used to train a machine learning model, then model accuracy is improved, but computational resources and computing power requirements increase
Solution Approach 1:
The patent extracts and removes redundant training samples from the dataset by comparing embeddings of training samples and identifying pairs with closely-matching embeddings. By eliminating these redundant samples while preserving diverse and informative ones, the method reduces training time and computational resource requirements while maintaining model accuracy on par with or better than training on the full dataset.
3Productivity
If redundant training samples are eliminated from the dataset, then training efficiency is improved, but model performance may deteriorate
Solution Approach 1:
The patent employs a feedback mechanism by monitoring class spread metrics and cross-class proximity metrics during the sample elimination process. The curation process dynamically adjusts which samples to remove based on these metrics, ensuring that model performance is maintained. This feedback-driven approach allows the system to eliminate redundant samples while preserving training samples that are critical for model performance, achieving both training efficiency and model performance goals.
Data Source
AI summary
The training dataset of a machine learning model is curated to eliminate redundant training samples from a supervised training dataset. The training samples are grouped into classes. An embedding of each training sample is used to search for pairs of training samples within a class having closely-matching embeddings. One training sample of the pair is eliminated. The search uses an approximate nearest neighbor search to find the redundant pairs. A curation process reduces the size of the training dataset to a user-defined removal rate or until the spread of the distribution of the training samples in each class and between classes meets a desired threshold.