Neural Network Inference via Split Quantization Levels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks in devices with limited resources, such as smartphones, face challenges in achieving low latency and low memory usage while maintaining accuracy, as extreme quantization can lead to reduced representation power and accuracy degradation.
Innovation Solution
A neural network-based inference method that involves receiving a quantization level for quantizing weights and activation values, generating quantized activation values, and performing inference using split activation values based on a ratio of quantization levels, allowing for efficient resource utilization and accuracy maintenance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If extreme quantization is applied to reduce memory usage and improve computational efficiency, then resource utilization improves, but representation power is limited causing accuracy degradation
Solution Approach 1:
The patent segments the quantization process by applying different quantization levels to different parts of the neural network. Specifically, it divides weights and activation values into multiple groups and applies different quantization precision to each group, allowing some parts to maintain high precision while others use lower precision, thus resolving the contradiction between memory usage and accuracy.
Solution Approach 2:
The patent implements local quality by assigning different quantization characteristics to different components of the neural network. Certain layers or channels that are more sensitive to precision requirements maintain higher quantization levels, while less sensitive parts use lower quantization levels, optimizing the balance between resource usage and accuracy preservation.
2Productivity
If extreme quantization is applied to improve computational efficiency, then processing speed improves, but representation power is limited causing accuracy degradation
Solution Approach 1:
The patent segments the computational process by dividing the neural network into multiple processing paths with different quantization levels. This allows the system to perform majority of computations at lower precision for efficiency while maintaining higher precision for critical operations, thus improving computational efficiency without severely compromising accuracy.
Solution Approach 2:
The patent introduces dynamic quantization where the quantization level can be adjusted based on the specific computation being performed, the layer being processed, or the importance of the data. This dynamic adjustment allows the system to optimize computational efficiency in real-time while preserving accuracy where needed.
Data Source
AI summary
Disclosed are a neural network-based training method, inference method and apparatus. The neural network-based inference method includes receiving a quantization level for quantizing a weight of a neural network and an activation value that is processed by the neural network, receiving a weight quantized based on the quantization level, generating a quantized activation value by quantizing the activation value based on the quantization level, and performing inference based on the quantized weight and the quantized activation value.