Hierarchical Scaling Factors for Neural Network Weight Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current optimizations for compressing and storing the weights of a trained model focus either on precision or range, failing to account for the varying needs of different network areas, particularly in generative networks where some parts require high precision while others do not.
Innovation Solution
Implementing a hierarchy of scaling factors that optimize both precision and range within neural networks, using quadtree-based region determination and microscaling (MX) formats to adaptively adjust weight compression based on the specific requirements of different network areas.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If uniform compression is applied to all weights in the neural network, then storage efficiency is improved, but precision requirements of different network areas are not met
Solution Approach 1:
The patent divides the neural network weights into different groups or regions (e.g., by layer, by magnitude, or by functional importance) and applies different compression scales to each group. This segmentation allows high-precision areas to maintain accuracy while low-precision areas achieve better compression, resolving the contradiction between uniform storage efficiency and variable precision requirements.
Solution Approach 2:
The patent implements local quality by assigning different precision levels to different parts of the network based on their specific requirements. Critical layers or connections receive higher precision (smaller scaling factors) while less critical areas use lower precision (larger scaling factors), optimizing the balance between overall storage efficiency and local precision needs.
2Manufacturing precision
If high precision is maintained for all weights, then network performance is improved, but storage and computational efficiency deteriorate
Solution Approach 1:
The patent introduces dynamic scaling factors that can be adjusted during training and inference based on the actual performance needs and resource constraints. This dynamic approach allows the system to adapt precision levels in real-time, maintaining high precision when needed for accuracy while reducing precision when computational efficiency is prioritized, thus resolving the contradiction between performance and efficiency.
Solution Approach 2:
The patent changes the precision parameter (scaling factor) based on the specific requirements of different network components. By varying this parameter across different layers, neurons, or weight magnitudes, the system achieves optimal balance between maintaining necessary precision and improving overall computational and storage efficiency.
3Quantity of substance
If compression is applied to weight ranges, then storage efficiency is improved, but precision in critical areas is compromised
Solution Approach 1:
The patent segments weights based on their magnitude ranges (e.g., small weights, medium weights, large weights) and applies different compression scaling factors to each range. This allows critical weights in specific ranges to maintain higher precision while less critical ranges achieve better compression, resolving the contradiction between storage efficiency and precision in critical areas.
Solution Approach 2:
The patent applies local quality by identifying critical areas or layers that require high precision and assigning them smaller scaling factors, while non-critical areas receive larger scaling factors for better compression. This localized precision strategy ensures that storage efficiency is improved overall without compromising the precision requirements of critical network components.
Data Source
AI summary
Embodiments described herein provide techniques to facilitate hierarchical scaling when quantizing neural network data to a reduced-bit representation. The techniques includes operations to load a hierarchical scaling map for a tensor associated with a neural network, partition the tensor into a plurality of regions that respectively include one or more subregions based on the hierarchical scaling map, hierarchically scale numerical values of the tensor based on a first scale factor and second scale factor via the matrix accelerator circuitry, the first scale factor based on a statistical measure of a subregion of numerical values of within a region of the plurality of regions and the second scale factor based on a statistical measure of the region that includes the subregion, and generate a quantized representation of the tensor via quantization of hierarchically scaled numerical values.