AI Model Quantization With Channel-Wise Shift Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for quantizing neural networks face accuracy loss and increased quantum parameter size, leading to degraded latency performance and errors, particularly in channel-wise quantization and cross-layer equalization.
Innovation Solution
An electronic apparatus employing a neural processing unit (NPU) that performs channel-wise shift scaling and layer-wise quantization, using shift scaling factors to maintain accuracy while reducing quantum parameter size and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If channel-wise quantization is performed to reduce quantization precision loss, then manufacturing precision is improved, but device complexity increases due to increased quantum parameter size
Solution Approach 1:
The patent segments the quantization process into two distinct stages: channel-wise quantization for precision and layer-wise quantization for compression. Channel-wise quantization is applied first to minimize precision loss by considering each channel's parameter distribution independently, then layer-wise quantization is applied to compress the quantum parameters to a manageable size (8 bits or less), resolving the contradiction between precision and complexity
Solution Approach 2:
The patent applies channel-wise quantization as a preliminary action before layer-wise quantization. By first performing channel-wise quantization to establish accurate quantum parameters based on actual parameter distributions, then subsequently applying layer-wise quantization for compression, the system preserves precision while achieving the desired compression ratio
2Manufacturing precision
If channel-wise quantization is performed to improve quantization accuracy, then manufacturing precision is improved, but loss of time increases due to increased parameter loading time
Solution Approach 1:
The patent segments quantum parameters into channel-wise parameters (stored in cache memory for fast access) and layer-wise parameters (stored in main memory). By separating the storage locations and access methods, the system can quickly load only the necessary channel-wise parameters from cache during inference, significantly reducing parameter loading time while maintaining quantization precision
Solution Approach 2:
The patent introduces cache memory as an intermediary between main memory and the computing units. Channel-wise quantum parameters are pre-loaded into cache memory, serving as a fast-access buffer that mediates between the large-capacity main memory and the speed requirements of the neural network computation, thereby reducing loading time
3Manufacturing precision
If cross-layer equalization is performed to adjust parameter ranges, then manufacturing precision is improved, but device complexity increases due to structural changes in neural network layers
Solution Approach 1:
The patent changes the parameters (scaling factors and zero-point values) of the neural network layers during the quantization process rather than modifying the network structure. By adjusting these parameters based on the actual parameter distributions of each layer, the system achieves alignment of parameter ranges and improved precision without introducing structural complexity
4Device complexity
If layer-wise quantization is performed to reduce quantum parameter size, then device complexity is reduced, but manufacturing precision deteriorates due to increased quantization error
Solution Approach 1:
The patent performs channel-wise quantization as a preliminary action before layer-wise quantization. The channel-wise quantization establishes accurate quantum parameters by considering each channel's parameter distribution independently, creating a high-precision baseline. The subsequent layer-wise quantization then compresses these parameters while referencing the established channel-wise parameters, thereby maintaining precision despite compression
Data Source
AI summary
An electronic apparatus is provided. The electronic apparatus includes a memory configured to store an artificial intelligence (AI) model including a plurality of layers and a processor, and the AI model may include a plurality of weight values that are scaled based on shift scaling factors different by a plurality of channels included in each of the plurality of layers and quantized by the plurality of layers, and the processor may, based on receiving input data, in a neural network computation process for the input data, compute a channel-wise computation result with an inverse-scaled composite scale parameter based on a shift scaling factor corresponding to each channel.