Neural Network Quantization With Hierarchical Tensor Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current optimizations for compressing and storing the weights of a trained neural network model focus either on precision or range, failing to address the varying needs of different network areas, particularly in generative networks where some parts require high precision while others do not.
Innovation Solution
Implementing a hierarchy of scaling factors that optimize both precision and range within neural network weights, allowing for tailored compression strategies based on the specific requirements of different network regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a single scaling factor is used for all weights in the neural network, then the compression process is simple and uniform, but it fails to optimize precision for different network regions, leading to suboptimal performance in generative networks
Solution Approach 1:
The patent divides the neural network weights into different groups or regions (e.g., by layer, by magnitude, or by functional importance) and applies different scaling factors to each group. This segmentation allows high-precision scaling for critical regions while using coarser scaling for less critical regions, resolving the contradiction between uniform simplicity and region-specific precision optimization.
Solution Approach 2:
The patent implements local quality by assigning different precision levels and scaling factors to different parts of the network based on their specific requirements. Critical layers or weight groups maintain higher precision with finer scaling factors, while non-critical regions use coarser scaling, thereby optimizing overall performance without uniformly high complexity.
2Measurement precision
If high precision scaling is applied to all weights, then the accuracy of the neural network is maintained, but the compression efficiency and resource utilization decrease
Solution Approach 1:
The patent applies high-precision scaling selectively to only those weight groups that require it for maintaining accuracy, rather than applying it universally. By identifying and prioritizing critical regions that need high precision while using coarser scaling for other regions, the system achieves necessary accuracy with improved compression efficiency and reduced resource consumption.
3Measurement precision
If different scaling factors are applied to different weight groups, then precision is optimized for specific network regions, but the complexity of the compression process increases
Solution Approach 1:
The patent organizes weights into distinct groups or layers and assigns scaling factors systematically to each group. This structured segmentation approach manages complexity by creating clear boundaries and rules for scaling factor assignment, making the multi-scale process more tractable and implementable compared to entirely uniform or entirely custom per-weight scaling.
Data Source
Figure 1
Figure 2A
Figure 2B~2C
AI summary
Embodiments described herein provide techniques to facilitate hierarchical scaling when quantizing neural network data to a reduced-bit representation. The techniques includes operations to load a hierarchical scaling map for a tensor associated with a neural network, partition the tensor into a plurality of regions that respectively include one or more subregions based on the hierarchical scaling map, hierarchically scale numerical values of the tensor based on a first scale factor and second scale factor via the matrix accelerator circuitry, the first scale factor based on a statistical measure of a subregion of numerical values of within a region of the plurality of regions and the second scale factor based on a statistical measure of the region that includes the subregion, and generate a quantized representation of the tensor via quantization of hierarchically scaled numerical values.