Data Augmentation Policies for Neural Network Training Without Separate Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for learning data augmentation policies in machine learning models are computationally expensive and inefficient, often requiring time-consuming search phases and training toy models on small datasets, leading to suboptimal results when applied to larger models and datasets.
Innovation Solution
A data augmentation system that determines optimal data augmentation policies in parallel with training the machine learning model, using a smaller search space and evaluating policies directly on the full training dataset, allowing for transferability across datasets and adapting transformation operations to the specific model and dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing systems perform a search phase to learn data augmentation policies, then the data augmentation policy can be optimized, but the process becomes time-consuming and computationally expensive
Solution Approach 1:
The patent applies preliminary action by determining optimal values for data augmentation hyperparameters in parallel with determining other model hyperparameters before training begins. This eliminates the need for a separate search phase, as the data augmentation policy is configured in advance based on preliminary analysis of the training data and model architecture, thereby reducing time loss while maintaining policy optimization quality
Solution Approach 2:
The patent changes parameters by focusing the search on a smaller, more relevant hyperparameter space specific to data augmentation (such as augmentation strength, probability, and sequence length) rather than searching the entire model configuration space. This targeted parameter approach maintains optimization effectiveness while significantly reducing computational cost and time
2Productivity
If toy machine learning models are trained on small training data sets to evaluate candidate policies, then the evaluation process is faster, but the results are suboptimal when applied to larger models and datasets
Solution Approach 1:
The patent uses copying by creating augmented versions of the training data through transformation operations (rotations, flips, color adjustments, etc.) rather than collecting new data. This allows the model to learn from multiple transformed copies of the same underlying patterns, effectively increasing data diversity and volume without requiring additional data collection or processing time
Solution Approach 2:
The patent applies dynamics by making the data augmentation policy adaptive during training. The hyperparameters for data augmentation are determined in parallel with model hyperparameters and can be adjusted based on training progress and performance feedback, allowing the augmentation strategy to evolve dynamically rather than remaining static throughout training
3Reliability
If a large search space of candidate augmentation policies is explored, then the optimal policy can be found, but the computational cost increases prohibitively
Solution Approach 1:
The patent segments the hyperparameter search space into distinct, manageable dimensions (augmentation type, strength, probability, sequence length) and determines optimal values for each independently in parallel. This segmentation transforms an intractable large-scale search into multiple smaller, concurrent searches, reducing computational cost while maintaining the ability to discover optimal policies
Solution Approach 2:
The patent creates a universal data augmentation framework that can be applied across different model architectures and datasets by determining hyperparameters in parallel with model-specific training. This multi-functional approach allows the same augmentation policy generation process to serve multiple purposes and contexts, reducing redundant computational effort
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model. One of the methods includes obtaining a training data set for training a machine learning model, the training data set comprising a plurality of training inputs; determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input; for each data augmentation policy, training the machine learning model using the data augmentation policy; determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy; and selecting a final data augmentation policy based using the quality measures of the machine learning models.