Classifier-Guided Dataset Reduction for Efficient AI Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of acquiring high-quality annotated datasets for machine learning is resource-intensive, particularly with large-scale and diverse data sources, leading to inefficiencies in computational resources and model accuracy due to redundant or irrelevant data.
Innovation Solution
A multi-stage process using CLIP and ViT models to convert datasets into high-dimensional vectors, followed by k-means clustering and a distribution classifier to identify a representative, non-redundant subset for training, ensuring similarity and diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large annotated dataset is used for training, then model accuracy is improved, but computational resources and training time increase significantly
Solution Approach 1:
The patent extracts a representative subset of data from the large annotated source dataset that is most similar to the target dataset. This subset extraction maintains model training effectiveness while reducing computational burden, directly resolving the contradiction between using large datasets for accuracy and reducing dataset size for efficiency.
Solution Approach 2:
The patent changes the parameter of dataset size by selectively reducing the annotated source dataset based on similarity metrics and redundancy analysis. This parameter transformation allows the system to operate with a smaller, optimized dataset that preserves accuracy while improving training efficiency.
2Reliability
If a large annotated dataset is used for training, then model performance is improved, but training time increases
Solution Approach 1:
The patent extracts only the most relevant and non-redundant portions of the annotated source dataset that are similar to the target dataset. This extraction process eliminates unnecessary data points that would consume training time without contributing to model performance, thus resolving the time-performance tradeoff.
Solution Approach 2:
The patent applies partial action by using only a subset of the available annotated data rather than the entire dataset. This partial approach is sufficient to achieve good model performance while significantly reducing training time, avoiding the excessive computation required for full dataset training.
3Quantity of substance
If redundant data is included in the training set, then data coverage is improved, but computational efficiency decreases
Solution Approach 1:
The patent changes the quality parameter of the dataset by removing redundant and irrelevant data points while preserving data coverage through similarity-based selection. This parameter optimization maintains comprehensive coverage of important data patterns while eliminating computational waste from redundant samples.
Solution Approach 2:
The system performs self-service by automatically identifying and removing redundant data through classifier guidance and similarity analysis. This self-optimization process ensures the training dataset maintains adequate coverage without manual intervention, improving computational efficiency while preserving necessary data diversity.
Data Source
AI summary
An example operation may include one or more of retrieving an annotated source dataset from a storage via a software application, retrieving a non-annotated target dataset from the storage via the software application, identifying a subset of data from the annotated source dataset, wherein the subset is configured to include source dataset data that is similar to the non-annotated target dataset, reducing the subset of data from the annotated source dataset by using a classifier to remove redundant data from the subset of data from the annotated source dataset, and classifying data from the non-annotated target dataset by a trained artificial intelligence (AI) model.