Neural Network Quantization Using Weighted Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network implementations require extensive computational and memory resources, limiting their use to expensive non-mobile environments and delaying result availability, which is addressed by optimizing quantization levels based on weighted entropy to reduce computational amounts while minimizing accuracy loss.
Innovation Solution
A neural network method and apparatus that adjusts quantization levels based on weighted entropy to optimize data processing, using techniques such as clustering and entropy-based logarithm data representation to minimize computational requirements and accuracy loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If neural network implementations use extensive computational operations on complex input data, then processing accuracy and reliability are improved, but computational resources and operational time requirements increase
Solution Approach 1:
The patent applies different quantization levels to different parts of the neural network data based on their importance. By calculating weighted entropy that considers both the frequency and magnitude of data values, the system assigns higher precision quantization to more important data elements and lower precision to less important ones, thereby maintaining processing accuracy while reducing overall computational resources required
Solution Approach 2:
The system dynamically adjusts quantization parameters (quantization levels) based on the statistical properties of the data. By computing weighted entropy from the data distribution and using this to determine optimal quantization levels, the system adapts the precision parameters to match the actual needs of the neural network operations, achieving efficient resource utilization
2Reliability
If neural network implementations use extensive computational operations, then processing accuracy is improved, but operational time is extended
Solution Approach 1:
The patent applies different quantization levels to different parts of the neural network data based on their importance. By calculating weighted entropy that considers both the frequency and magnitude of data values, the system assigns higher precision quantization to more important data elements and lower precision to less important ones, thereby maintaining processing accuracy while reducing overall computational resources required
Solution Approach 2:
The system performs quantization optimization as a preliminary step before actual neural network execution. By pre-calculating the weighted entropy and determining optimal quantization levels in advance, the system prepares efficiently optimized data representations that will speed up subsequent operations without sacrificing accuracy
3Productivity
If standard quantization methods are used, then computational resources are reduced, but accuracy loss increases
Solution Approach 1:
The system dynamically adjusts quantization parameters (quantization levels) based on the statistical properties of the data. By computing weighted entropy from the data distribution and using this to determine optimal quantization levels, the system adapts the precision parameters to match the actual needs of the neural network operations, achieving efficient resource utilization
Solution Approach 2:
The weighted entropy calculation provides feedback about the data distribution and importance characteristics. This feedback is used to iteratively determine optimal quantization levels that balance between reducing computational resources and maintaining processing accuracy, ensuring that quantization does not unnecessarily degrade model performance
Data Source
AI summary
Provided are a neural network method and an apparatus, the method including obtaining a set of floating point data processed in a layer included in a neural network, determining a weighted entropy based on data values included in the set of floating point data, adjusting quantization levels assigned to the data values based on the weighted entropy, and quantizing the data values included in the set of floating point data in accordance with the adjusted quantization levels.