ANN Training Data Subsampling Using Perplexity Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training artificial neural networks (ANNs) is computationally intensive, particularly for large datasets, consuming significant resources like memory, processor cycles, and power, and existing methods to reduce training time often compromise classification performance.
Innovation Solution
Subsampling training data during ANN training by computing perplexity scores to identify and remove samples that are either too certain or too uncertain, using metrics like Kullback-Leibler divergence, and periodically reintroducing removed samples to maintain accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all training data samples are used during ANN training, then model accuracy is maintained, but training time and resource consumption increase significantly
Solution Approach 1:
The patent extracts and removes certain training samples from the training dataset based on perplexity score thresholds. Samples with scores below the threshold (indicating the model is either too certain or too uncertain) are excluded from training, allowing the model to achieve comparable accuracy with fewer samples, thus reducing training time and computational resources.
Solution Approach 2:
The patent dynamically adjusts the composition of the training dataset by changing the parameter of sample inclusion based on perplexity scores. By modifying which samples are included (those with scores between threshold values) and excluding others, the training process optimizes the balance between accuracy and training efficiency.
2Measurement precision
If all training data samples are used during ANN training, then model accuracy is maintained, but computing resource consumption increases
Solution Approach 1:
The patent extracts and removes certain training samples from the training dataset based on perplexity score thresholds. Samples with scores below the threshold (indicating the model is either too certain or too uncertain) are excluded from training, allowing the model to achieve comparable accuracy with fewer samples, thus reducing training time and computational resources.
Solution Approach 2:
The patent applies partial action by training on only a subset of the training data rather than the complete dataset. By selectively including only samples with perplexity scores within the threshold range, the method uses a partial portion of the data that is sufficient to achieve the desired accuracy while consuming fewer computing resources.
3Productivity
If training data samples with extreme perplexity scores are removed, then training efficiency improves, but model accuracy may deteriorate
Solution Approach 1:
The patent implements feedback by periodically evaluating the model's performance and adjusting the perplexity score thresholds accordingly. The system monitors whether removing samples with extreme perplexity scores maintains the desired accuracy level, and can adjust the inclusion/exclusion criteria based on observed performance, ensuring that training efficiency improvements do not compromise model accuracy.
Solution Approach 2:
The patent dynamically adjusts the composition of the training dataset by changing the parameter of sample inclusion based on perplexity scores. By modifying which samples are included (those with scores between threshold values) and excluding others, the training process optimizes the balance between accuracy and training efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Perplexity scores are computed for training data samples during ANN training. Perplexity scores can be computed as a divergence between data defining a class associated with a current training data sample and a probability vector generated by the ANN model. Perplexity scores can alternately be computed by learning a probability density function ("PDF") fitting activation maps generated by an ANN model during training. A perplexity score can then be computed for a current training data sample by computing a probability for the current training data sample based on the PDF. If the perplexity score for a training data sample is lower than a threshold, the training data sample is removed from the training data set so that it will not be utilized for training during subsequent epochs. Training of the ANN model continues following the removal of training data samples from the training data set.