Neural Network Layer Parameter Resolution Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face challenges in efficiently using memory and energy resources while maintaining performance, particularly in applications requiring low resolution parameters and activations.
Innovation Solution
A machine learning system is designed with layers connected to multiple preceding layers, where parameters and activations have lower resolutions, specifically with reduced quantization error, allowing for efficient memory usage and reduced computing requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If parameters and activations are stored with high resolution, then performance and accuracy are maintained, but memory space requirements increase
Solution Approach 1:
The patent applies different quantization resolutions to different layers based on their connectivity. Layers connected to multiple preceding layers use lower resolution (higher quantization) while layers connected to fewer layers maintain higher resolution. This local differentiation optimizes memory usage without uniformly compromising precision across the entire network.
Solution Approach 2:
The patent changes the resolution parameter of parameters and activations based on layer connectivity. By dynamically adjusting the number of bits used to represent parameters and activations according to their position in the network and connectivity patterns, the system achieves efficient memory utilization while maintaining necessary precision where required.
2Use of energy by moving object
If parameters are strongly quantized to reduce memory usage, then computing requirements are reduced, but performance may deteriorate
Solution Approach 1:
Different layers are assigned different quantization strengths based on their input connectivity. Layers receiving inputs from multiple preceding layers apply stronger quantization (lower resolution) while layers with fewer inputs maintain weaker quantization (higher resolution). This localized approach reduces overall computing energy without uniformly degrading performance across all layers.
Solution Approach 2:
The neural network is segmented into groups based on connectivity patterns, with each group receiving a tailored quantization level. This segmentation allows the system to apply aggressive quantization only where it has minimal impact on performance, while preserving precision in critical pathways.
3Measurement precision
If uniform high resolution is used across all layers, then performance is maintained, but memory and energy efficiency are reduced
Solution Approach 1:
The patent implements non-uniform resolution allocation where each layer's precision is optimized according to its specific connectivity pattern. Layers connected to multiple preceding layers use lower resolution representations, while layers with fewer connections maintain higher resolution. This local quality differentiation maximizes processing efficiency without sacrificing overall system performance.
Data Source
AI summary
A machine learning system, in particular a deep neural network. The machine learning system includes a plurality of layers that are connected to one another. The layers each ascertain an output variable as a function of an input variable and at least one parameter that is stored in a memory. The parameters of those layers that are connected to a further, in particular preceding, layer are each stored in the memory using a higher resolution than the parameters of those layers that are connected to a plurality of further, in particular preceding, layers. In addition, A method, a computer program, and a device for creating the machine learning system, are described.