Machine Learning Speed-Up via Layer Variance Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning speed-up techniques, such as gradient skip technology, often compromise accuracy by skipping machine learning layers, leading to incomplete target accuracy in machine learning models, especially in deep neural networks.
Innovation Solution
A method that calculates the variance of weight information for each layer in a machine learning model at predetermined intervals to determine which layers can safely skip learning, thereby reducing unnecessary processing and improving accuracy by setting thresholds based on variance peaks rather than weight or weight gradient values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If gradient skip technology is used to skip machine learning in certain layers, then processing time is reduced and speed-up is achieved, but machine learning accuracy is deteriorated and target accuracy cannot be attained
Solution Approach 1:
The patent changes the parameter used for determining skip targets from fixed weight thresholds to dynamic variance metrics. By calculating variance of weight information at predetermined intervals and comparing it to variance peak values, the system dynamically identifies layers suitable for skipping based on actual learning progress rather than static criteria, thus maintaining accuracy while achieving speed-up
Solution Approach 2:
The patent implements feedback mechanisms by continuously monitoring variance of weight information during the learning process. The system calculates variance at predetermined intervals, compares current variance to peak variance values, and uses this feedback to dynamically determine which layers can be skipped. This feedback loop ensures that skipping decisions are based on real-time learning state, preventing accuracy deterioration
2Manufacturing precision
If machine learning is performed frequently with high learning rate to update new learning portions, then learning accuracy is improved, but processing time increases due to unnecessary aggregation and reflection processing
Solution Approach 1:
The patent extracts and identifies specific layers that have reached learning saturation by monitoring variance metrics. Once a layer's variance falls below the threshold determined from peak variance, that layer is extracted from the active learning process and marked for skipping. This allows the system to maintain high learning rates for layers that still need training while eliminating unnecessary processing in layers that have converged
Solution Approach 2:
The patent segments the machine learning model into different layers and applies different processing strategies to each segment. By calculating variance for each layer independently and comparing to layer-specific peak variance values, the system can selectively skip individual layers or groups of layers while continuing full learning processing for other layers, thus reducing overall processing time without compromising accuracy in layers that still require training
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method of a speed-up processing, the method including: calculating variance of weight information regarding a weight updated by machine learning, for each layer included in a machine learning model at a predetermined interval at time of the machine learning of the machine learning model; and determining a suppression target layer that suppresses the machine learning on the basis of a peak value of the variance calculated at the predetermined interval and the variance of the weight information calculated at the predetermined interval.