Neural Network Processor Layer Interruption for Transfer Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network systems require significant time for learning, particularly in transfer learning, due to the need for multiple iterations across all layers, including those where learning may be saturated and non-productive.
Innovation Solution
A neural network system and method that determines which layers to interrupt learning in by comparing weight distributions between iterations, allowing subsequent learning iterations to focus only on non-saturated layers, thereby reducing overall learning time and increasing transfer learning speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If learning is performed on all layers for multiple iterations, then learning completeness is improved, but learning time increases significantly
Solution Approach 1:
The patent segments the learning process by layer, dividing all layers into multiple groups based on weight distribution characteristics. Different groups undergo different numbers of learning iterations, with saturated layers processed in fewer iterations and non-saturated layers receiving more iterations. This segmentation allows the system to maintain learning completeness while reducing overall learning time by avoiding unnecessary iterations on already-saturated layers.
Solution Approach 2:
The patent applies partial action by performing learning iterations selectively rather than uniformly across all layers. Layers are categorized into groups where only certain groups undergo full learning iterations while others receive reduced or no iterations based on their saturation status. This partial approach eliminates wasted computational effort on saturated layers while ensuring adequate learning on non-saturated layers.
2Productivity
If learning iterations are reduced to save time, then learning speed is improved, but learning accuracy deteriorates
Solution Approach 1:
The patent applies local quality by assigning different learning iteration counts to different layer groups based on their specific saturation characteristics. Rather than uniformly reducing iterations across all layers, the system identifies which layers have achieved sufficient learning (saturated) and which require more iterations (non-saturated). This localized approach ensures that learning accuracy is maintained for critical layers while accelerating the overall process by reducing iterations on already-learned layers.
Solution Approach 2:
The patent implements feedback mechanisms by continuously monitoring weight distribution across layers and using this information to dynamically adjust the learning process. The controller evaluates weight distribution after each iteration and uses this feedback to determine which layers should continue learning and which can be interrupted. This feedback-driven approach ensures that learning accuracy is maintained by continuing iterations on non-saturated layers while enabling early termination on saturated layers to improve learning speed.
3Measurement precision
If weight distribution comparison is performed for each layer, then layer selection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by performing weight distribution comparison and layer classification before the main learning iterations begin. The controller initially evaluates all layers to determine their saturation status and assigns them to different groups. This preliminary classification establishes a roadmap for subsequent learning iterations, avoiding the need for continuous complex comparisons during the learning process and reducing overall computational complexity while maintaining high layer selection accuracy.
Data Source
AI summary
A neural network system includes a processor and a memory. The processor is configured to perform learning including multiple learning iterations on multiple layers, to determine at least one layer in which the learning is interrupted among the multiple layers. The determination of the at least one layer in which the learning is interrupted is based on a result of comparing for each of the multiple layers a distribution of first weight values resulting from a first learning iteration with a distribution of second weight values resulting from a second learning iteration. The processor is also configured to perform a third learning iteration in layers except the at least one layer for which interruption of the learning has been determined. The memory stores first distribution information of the first weight values and second distribution information of the second weight values and is configured to provide the first distribution information and the second distribution information to the processor when the second learning iteration is completed.