Neural Network Quantization Scale Factor Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for quantizing neural network weights and activations during training introduce errors due to shared scale factors across tensor dimensions, leading to reduced accuracy and increased processing times when dealing with transposed matrices.
Innovation Solution
Applying different scale factors to sub-matrices along multiple dimensions of tensors, allowing the same scale factor to be used for both a matrix and its transposed version, thereby reducing the number of read operations and improving processing efficiency during neural network training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If different scale factors are applied to different vectors along a single dimension of the weight tensor and/or activation tensor, then the accuracy of the deep neural network is improved, but the processing time is substantially increased
Solution Approach 1:
The patent segments the tensor into multiple groups along a first dimension, where each group shares a common scale factor. This segmentation approach balances between using fine-grained scale factors (for accuracy) and coarser grouping (for efficiency), resolving the contradiction by organizing data into manageable segments that can be processed with shared parameters.
Solution Approach 2:
The patent combines multiple vectors along the first dimension into groups that share a common scale factor. By merging vectors that can tolerate the same scaling parameter, the system reduces the total number of scale factors needed, thereby decreasing memory read operations and processing time while maintaining acceptable quantization accuracy.
2Productivity
If the same scale factor is shared across the many dimensions of a given tensor, then the processing time is reduced, but the precision of the individual elements within a scaled tensor is reduced
Solution Approach 1:
The patent applies local quality by allowing different scale factors for different groups of vectors along the first dimension, while maintaining a single scale factor for all vectors along the second dimension. This localized differentiation provides just enough precision where needed (along the first dimension) while maintaining processing efficiency along the second dimension.
Solution Approach 2:
The patent introduces a dimensional distinction by applying scale factors differently along the first dimension versus the second dimension of the tensor. This dimensional approach allows the system to optimize precision along one dimension while maintaining efficiency along another, effectively navigating the precision-efficiency tradeoff through dimensional differentiation.
3Reliability
If different scale factors are applied to different vectors along a single dimension of the weight tensor and/or activation tensor, then the amount of error introduced by quantization is reduced, but the number of read operations from memory is substantially increased
Solution Approach 1:
The patent segments vectors into groups along the first dimension, where each group shares a scale factor. This segmentation reduces the total number of scale factors from one-per-vector to one-per-group, significantly reducing memory read operations while still providing group-level precision control to maintain quantization reliability.
Solution Approach 2:
The patent makes scale factors universal at the group level, where a single scale factor serves multiple vectors within a group. This multi-functionality approach allows one scale factor to fulfill the precision needs of multiple vectors simultaneously, reducing the total number of read operations while maintaining adequate quantization quality across all vectors in the group.
Data Source
AI summary
One embodiment of a computer-implemented method for processing a neural network comprises receiving a first quantized matrix that corresponds to a portion of a multi-dimensional input tensor and has been quantized based on a first scale factor; and performing one or more computational operations using the first quantized matrix and the first scale factor to generate one or more data values that correspond to a first portion of a multi-dimensional output tensor.