Dynamic Neural Network Weight Precision Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network models require substantial processing and memory resources, leading to high monetary and resource costs, especially when implemented across many physical computing units. Additionally, representing weights with lower precision improves resource conservation but reduces model and training quality.
Innovation Solution
The technology dynamically adjusts the precision of weights in a neural network model based on their influence on the model's output and their fluctuations during training, allowing for individualized and dynamic precision levels. This is achieved by categorizing weights into different memory blocks based on computed heuristics and storing them using varying numbers of memory bits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weights are stored with lower precision to conserve memory resources, then memory usage is reduced, but model and training quality deteriorates
Solution Approach 1:
The patent applies local quality by assigning different precision levels to different weights based on their individual characteristics. Weights are categorized into groups (e.g., first group with higher precision, second group with lower precision) according to their influence on model output and training dynamics. This allows critical weights to maintain high precision while less critical weights use lower precision, optimizing the trade-off between memory efficiency and model quality.
Solution Approach 2:
The patent dynamically changes the precision parameter of weight storage during the training process. The system monitors training progress and adjusts the precision allocation for different weight groups based on their convergence behavior and importance. This enables adaptive precision management that responds to actual training needs rather than using a fixed precision level throughout training.
2Manufacturing precision
If uniform high precision is used for all weights, then model quality is maintained, but memory resources and processing costs increase substantially
Solution Approach 1:
Instead of applying uniform precision across all weights, the patent implements local quality by differentiating precision requirements for individual weights or weight groups. Each weight is evaluated based on its contribution to model output and training stability, allowing the system to allocate high precision only where necessary and lower precision where acceptable, thereby reducing overall memory consumption while maintaining model quality.
Solution Approach 2:
The patent segments the weight parameters into multiple groups with different precision requirements. By dividing the weight matrix into segments (e.g., important weights vs. less important weights), the system can apply different storage precision levels to each segment, optimizing the balance between memory efficiency and model performance rather than treating all weights uniformly.
3Productivity
If deep neural networks are implemented across many physical computing units, then processing capability is enhanced, but monetary and resource costs increase
Solution Approach 1:
The patent changes the precision parameter of weight representation to reduce the resource footprint of deep neural networks. By using mixed precision (combining higher and lower precision formats) for different weights, the system reduces memory bandwidth requirements and storage needs, thereby lowering the number of physical computing units needed to achieve the same processing capability, which directly reduces monetary and resource costs.
Data Source
AI summary
Examples of the presently disclosed technology provide computerized systems and methods for dynamically adjusting amounts of precision (i.e., numbers of bits) used to represent and store individual weights of a neural network model (e.g., a DNN model) in response to training. Examples can use various heuristics to intelligently determine these individualized, and dynamic precision levels. For instance, a heuristic may include one or more of: (1) a measurement quantifying a magnitude of a respective weight's influence on output of the neural network model during a most recent set of training iterations (weights having relatively higher influence can be represented using higher precision); and (2) a measurement quantifying a magnitude of the respective weight's fluctuations in value during the most recent set of training iterations (weights having relatively smaller fluctuations in value can be represented using higher—i.e., more granular—precision).