Adaptive Data Sampling for AI Model Training Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large AI models on edge devices with limited computational power is hindered by high memory consumption and computational complexity, limiting their application in various scenarios due to the need for massive datasets and extensive training times.
Innovation Solution
A method that calculates importance metrics for data samples based on previous training epochs, determines sampling probabilities, and selects a subset of samples for training, allowing for accelerated AI model training without a pre-training stage and without removing any data samples, using only the differences in model outputs between epochs to measure sample importance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If massive datasets are used to train large AI models, then model accuracy is improved, but training time and computational complexity increase significantly
Solution Approach 1:
The patent applies partial action by selecting a subset of training data samples based on importance metrics rather than using the entire massive dataset. The system calculates importance scores for each sample and selectively trains on high-importance samples, achieving acceptable model accuracy with reduced training time and computational resources.
Solution Approach 2:
The patent changes the parameter of data sample selection from uniform random sampling to importance-based sampling. By introducing importance metrics that evaluate the contribution of each data sample to model learning, the system transforms the training process to focus on high-value samples, thereby reducing training time while maintaining accuracy.
2Adaptability or versatility
If large AI models with millions or billions of parameters are trained, then model capability is improved, but memory consumption and computational complexity increase
Solution Approach 1:
The patent reduces computational complexity by applying partial action - training large models on a selected subset of important data samples rather than processing the entire dataset. This approach maintains model capability while significantly reducing the computational burden during training.
Solution Approach 2:
The patent applies local quality by differentiating between high-importance and low-importance data samples. The system assigns different weights or selection probabilities to samples based on their local importance metrics, allowing the model to focus computational resources on samples that provide the most learning value.
3Productivity
If data samples are removed to reduce training time, then training efficiency is improved, but model accuracy may deteriorate
Solution Approach 1:
The patent changes the selection parameter from random uniform distribution to importance-based distribution. By calculating importance metrics for each data sample and using these metrics to determine selection probability, the system ensures that high-importance samples are retained while low-importance samples are reduced, maintaining accuracy while improving efficiency.
Solution Approach 2:
The patent implements feedback by continuously evaluating the importance of data samples based on model predictions and training progress. The importance metrics are updated during training, allowing the system to adaptively adjust which samples are selected, ensuring that samples critical for accuracy are not removed while enabling efficient training on less critical samples.
Data Source
AI summary
A method has the steps of: calculating importance metrics of a plurality of data samples based on predictions of an artificial-intelligence (AI) model obtained from the plurality of data samples in a plurality of previous training epochs without using labels of the plurality of data samples and without using a learning rate of the AI model; calculating sampling probabilities of the plurality of data samples based on the importance metrics thereof; selecting a subset of the plurality of data samples based on the sampling probabilities of the of plurality of data samples; and training the AI model using the selected subset of the plurality of data samples for one or more epochs.