Neural Network Training With Selective High-Loss Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for training neural networks are inefficient and require a large number of operations, leading to prolonged training times.
Innovation Solution
A computing system is configured to train neural networks by processing batches of samples through three phases: forward pass, backward pass, and optimization, where a defined proportion of loss values with the highest values are dropped during the forward pass, and gradients are calculated based on the average of these values, reducing the number of operations required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional methods process all loss values during training, then training accuracy is maintained, but training time and computational operations increase significantly
Solution Approach 1:
The patent extracts and removes a defined proportion of loss values (e.g., lowest 50% of loss values) from the training computation process. By dropping these loss values during the forward pass and only computing gradients for the remaining high-loss samples, the system reduces computational operations while maintaining effective training, directly resolving the contradiction between training speed and training time
Solution Approach 2:
Instead of processing all loss values equally, the patent applies partial action by computing gradients only for a subset of loss values (those above a threshold or top proportion). This selective processing reduces the number of operations required while still achieving effective model convergence, thereby improving productivity without excessive loss of training time
2Reliability
If all loss values are used for gradient calculation, then comprehensive model updating is achieved, but computational complexity and operations increase
Solution Approach 1:
The patent extracts only the necessary loss values (high-loss samples) for gradient calculation, removing redundant low-loss samples from computation. This extraction maintains training effectiveness by focusing on informative samples while reducing computational complexity in the backward pass
Solution Approach 2:
The patent applies different processing quality to different loss values: high-loss samples receive full gradient computation while low-loss samples are dropped. This local differentiation in computation quality maintains reliability for critical samples while reducing overall computational complexity
3Manufacturing precision
If batches of samples are processed through complete forward and backward passes, then thorough training is achieved, but the number of operations becomes prohibitively large
Solution Approach 1:
The patent applies partial action by processing only a subset of batches through complete forward and backward passes. By dropping certain loss values and computing gradients selectively, the system maintains training precision for critical samples while improving productivity through reduced operations per second
Data Source
AI summary
Embodiments of the present disclosure include systems and methods for reducing operations for training neural networks. A plurality of training data selected from a training data set is used as a plurality of inputs for training a neural network. The neural network includes a plurality of weights. A plurality of loss values are determined based on outputs generated by the neural network and expected output data of the plurality of training data. A subset of the plurality of loss values are determined. An average loss value is determined based on the subset of the plurality of loss values. A set of gradients is calculated based on the average loss value and the plurality of weights in the neural network. The plurality of weights in the neural network are adjusted based on the set of gradients.