Neural Network Training Data Sizing via Shrinking Estimates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large machine learning models require significant computational resources, incur high latency, and have a substantial environmental impact during inference, making it challenging to deploy them efficiently.
Innovation Solution
A training system that uses a model shrinking mapping to determine the required amount of training data for a smaller machine learning model, allowing it to match the performance of a larger model while reducing computational costs, latency, and environmental impact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large machine learning model is used to achieve high performance on machine learning tasks, then the model accuracy and capability are improved, but the computational cost, latency, and environmental impact during inference increase significantly
Solution Approach 1:
The patent changes the parameter of model size by training a smaller model with modified architecture parameters (e.g., reduced hidden layer dimensions, fewer layers) while adjusting training data parameters (increased data quantity, augmented data diversity) to compensate, thereby achieving comparable performance with lower computational cost
Solution Approach 2:
The patent applies local quality by selectively modifying specific components of the neural network architecture (e.g., reducing complexity in certain layers while maintaining others) rather than uniformly scaling down the entire model, allowing targeted optimization of computational efficiency while preserving critical performance capabilities
2Reliability
If a large machine learning model is deployed to achieve high performance, then the model capability is improved, but the inference latency increases
Solution Approach 1:
The patent modifies architectural parameters such as reducing the number of layers, decreasing hidden unit counts, and optimizing activation functions to accelerate inference speed, while compensating for performance loss through increased training data and adjusted loss functions
3Use of energy by moving object
If a smaller machine learning model is trained to reduce computational cost, then the deployment efficiency is improved, but the model performance may deteriorate
Solution Approach 1:
The patent applies partial or excessive action by training the smaller model on an excessive amount of training data beyond what a standard model would require, and using data augmentation techniques to generate additional training examples, thereby compensating for the reduced model capacity and maintaining performance
Solution Approach 2:
The patent performs preliminary action by conducting extensive hyperparameter tuning, architecture search, and training data preparation before final model training, including pre-processing training data with augmentation techniques and selecting optimal architecture configurations through preliminary experiments
4Object-generated harmful factors
If a smaller machine learning model is used to reduce environmental impact, then the sustainability is improved, but the model capability may be reduced
Solution Approach 1:
The patent changes model parameters by reducing architecture size (fewer parameters, smaller layers) to decrease environmental impact, while compensating for capability loss through increased training data volume, data augmentation, and optimized training procedures
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a size of a training data set for training a machine learning model. In one aspect, the size is determined using a shrinking estimate that estimates how much training data is needed to train a smaller machine learning model to achieve the same performance as a larger machine learning model.