Machine Learning Batch Typicality via Distribution Shuffling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine-learned models trained with imbalanced batches, known as atypical batches, experience negative impacts on training quality, speed, and stability due to larger variances in gradients, and existing solutions like reduced learning rate and gradient clipping introduce biases and inaccuracies.
Innovation Solution
A computer-implemented method that determines atypical batches by analyzing batch distributions using existing representations of training data, such as outputs from a prior machine-learned model, and shuffles training examples across batches to create more balanced batches, improving typicality and reducing computational resource wastage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If training examples are separated into batches for machine learning training, then training efficiency is improved, but batch imbalances occur leading to negative impacts on model training quality and stability
Solution Approach 1:
The patent changes the distribution parameters of training examples within batches by shuffling examples between batches. This modifies the batch composition parameters to achieve more balanced class distributions across all batches, thereby improving training stability while maintaining the batched training efficiency.
Solution Approach 2:
The patent performs preliminary analysis of batch distributions before actual model training begins. By identifying atypical batches in advance and redistributing examples beforehand, the system prevents training instability issues from occurring during the main training process, thus maintaining both efficiency and reliability.
2Reliability
If reduced learning rate and gradient clipping are used to handle atypical batches, then model training stability is improved, but biases and inaccuracies are introduced
Solution Approach 1:
The patent extracts and removes atypical batches from the training process by identifying them through distribution analysis and redistributing their examples to typical batches. This eliminates the need for corrective measures like gradient clipping that would otherwise introduce biases, thereby maintaining both stability and accuracy.
Solution Approach 2:
The patent converts the harmful effect of atypical batches into a benefit by using their examples to fill gaps in other batches. Instead of discarding or correcting atypical batches through biased methods, the system redistributes their examples to create more balanced batches, turning a training problem into a solution that improves overall batch quality.
3Productivity
If computational resources are used to determine gradients for atypical batches, then model training proceeds, but computational resource wastage occurs
Solution Approach 1:
The patent performs preliminary identification and redistribution of atypical batch examples before gradient computation begins. By correcting batch imbalances in advance, the system ensures that computational resources are not wasted on calculating gradients for atypical batches, thereby improving resource efficiency without hindering training progress.
Data Source
AI summary
Systems and methods described herein can improve typicality of batches for machine learning. The systems and methods can include obtaining a corpus of training data, the corpus of training data including one or more training examples. The systems and methods can include generating a first batch set including a plurality of batches from the corpus of training data, each of the batches including a subset of the one or more training examples. The systems and methods can include determining a batch distribution of a first batch of the first batch set. The systems and methods can include determining that the first batch is an atypical batch based on the batch distribution of the first batch. The systems and methods can include, in response to determining that the first batch is an atypical batch, shuffling the training examples of the first batch and one or more second batches of the first batch set to generate a second batch set. The systems and methods can include training a first machine-learned model using the second batch set.