Data Augmentation Policy Training Without Separate Search Phases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for learning data augmentation policies in machine learning models are computationally expensive and inefficient, often requiring time-consuming search phases and training toy models on small datasets, leading to suboptimal results when applied to larger models and datasets.
Innovation Solution
A data augmentation system determines optimal data augmentation policies directly on the full training dataset, evaluating candidate policies in parallel with model training, reducing the search space and eliminating the need for toy models, and allowing policies to be transferrable across datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing systems perform a search phase before training to learn data augmentation policies, then they can find candidate augmentation policies, but the process becomes time-consuming and computationally expensive
Solution Approach 1:
The patent merges the data augmentation policy search phase with the model training phase into a single unified process. Instead of performing search before training, the system learns augmentation policies and trainsthe model simultaneously, eliminating the time loss associated with sequential execution while maintaining policy quality through joint optimization.
Solution Approach 2:
The patent introduces preliminary action by using a smaller proxy model to pre-explore the data augmentation policy search space before training the full model. This preliminary exploration on a scaled-down version provides informed initialization for the main training process, reducing the overall search time and computational cost.
2Use of energy by moving object
If existing systems train toy models on small datasets to learn data augmentation policies, then they can reduce computational costs, but the policies become suboptimal when applied to larger models and datasets
Solution Approach 1:
The patent applies dynamics by implementing a progressive training approach where the model size and dataset scale dynamically during the learning process. The system starts with smaller configurations to explore the policy space efficiently, then progressively scales up to larger models and datasets, allowing the policy to adapt and maintain effectiveness across different scales while managing computational costs.
Solution Approach 2:
The patent uses a proxy model as an intermediary between the computational constraints of small-scale training and the requirements of large-scale deployment. This intermediary model enables policy learning on manageable computational resources while maintaining transferability to larger models through careful design of the proxy model's relationship to the target model.
3Measurement precision
If existing systems search a large search space of candidate augmentation policies, then they can find optimal policies, but the search becomes prohibitively expensive
Solution Approach 1:
The patent segments the large search space of data augmentation policies into smaller, manageable subspaces based on policy characteristics and model requirements. By dividing the search space, the system can explore each segment more efficiently using targeted search strategies, reducing the overall computational expense while still achieving optimal or near-optimal policies through systematic coverage of all segments.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model. One of the methods includes obtaining a training data set for training a machine learning model, the training data set comprising a plurality of training inputs; determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input; for each data augmentation policy, training the machine learning model using the data augmentation policy; determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy; and selecting a final data augmentation policy based using the quality measures of the machine learning models.