Neural Network Input Selection Using Loss-to-Gradient Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks (DNNs) is resource-intensive and time-consuming due to the need for optimizing thousands or billions of weights, requiring efficient techniques to optimize the training process.
Innovation Solution
A training system computes scores for input training items based on gradients of a loss function with respect to time, preferentially selecting items with higher scores for subsequent batches, using GPUs for computations and estimating gradients through linear regression, and employing a scoring methodology to select inputs for batches and validation sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all input training items are propagated through the network uniformly, then the training process is simple to implement, but the training time is excessive and resource consumption is high
Solution Approach 1:
The patent applies local quality by differentiating the treatment of individual training inputs based on their specific characteristics. Each input is assigned a score reflecting its importance or difficulty, and this local differentiation guides the selective propagation process, ensuring that resources are concentrated on inputs that benefit most from additional training passes.
Solution Approach 2:
The patent implements continuity of useful action by maintaining a continuous training process where inputs are repeatedly propagated through the network in multiple epochs. Rather than completing one full pass through all inputs and then stopping, the system continuously selects and propagates high-scoring inputs, maintaining productive training action throughout the process.
2Manufacturing precision
If training inputs are selected uniformly from the training set, then the selection process is simple, but the convergence speed to optimal parameters is slow
Solution Approach 1:
The patent applies dynamics by making the training input selection process adaptive rather than static. The selection probabilities of training inputs are dynamically adjusted based on their scores, which are computed based on the current state of network training. This dynamic reweighting allows the system to focus on difficult inputs when the network is struggling and gradually shift to easier inputs as training progresses.
Solution Approach 2:
The patent implements feedback by using the training error or loss associated with each input to compute selection scores. This feedback loop allows the system to identify which inputs are causing the most difficulty and prioritize them for additional training passes, thereby accelerating convergence by addressing the most problematic cases first.
3Reliability
If the network is trained on all training items repeatedly, then comprehensive coverage is achieved, but computational resources are wasted on items the network already handles well
Solution Approach 1:
The patent applies partial action by selectively propagating only a subset of training inputs through the network in each epoch, rather than processing all inputs uniformly. Inputs with higher scores (indicating greater difficulty or importance) are selected for propagation, while inputs that the network already handles well are skipped or given lower priority, reducing unnecessary computational expenditure.
Solution Approach 2:
The patent implements parameter changes by modifying the selection probabilities of training inputs based on their computed scores. Rather than using fixed uniform sampling, the system dynamically adjusts which inputs are propagated and how many times, changing the effective training parameters adaptively to optimize resource utilization while maintaining training effectiveness.
Data Source
AI summary
Some embodiments provide a method for training a machine-trained network that includes multiple parameters. The method propagates a batch of input training items through the network to generate output values and compute values of a loss function for each of the input training items. The method uses the computed values of the loss function for the input training items to adjust the parameters of the network. The method computes a gradient of the loss function for each of the input training items. The method selects input training items for subsequent batches of input training items based on a ratio of the value of the loss function to the gradient of the loss function for each of the input training items.