Neural Network Quantization Parameter Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Neural Networks (NNs) face challenges in efficiently representing values in hardware due to the trade-off between accuracy and resource efficiency, particularly when using fixed point number formats, which can lead to reduced accuracy with fewer bits and increased complexity with floating point formats.
Innovation Solution
A computer-implemented method and device for identifying quantization parameters that transform values into optimal fixed point number formats by determining a cost metric combining error and implementation costs, back-propagating gradients, and adjusting bit widths to minimize the cost metric, allowing for efficient hardware implementation while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed point number formats with fewer bits are used to represent values in hardware, then resource efficiency is improved, but accuracy deteriorates
Solution Approach 1:
The patent applies dynamics by making the bit width of fixed point numbers adjustable rather than static. The system dynamically determines optimal bit widths for different data types (activations, weights, biases) during model compilation, allowing the representation precision to be adapted to the specific requirements of each layer and operation, thus resolving the trade-off between resource efficiency and accuracy
Solution Approach 2:
The patent changes the parameter of bit width for fixed point number representation. By systematically exploring different bit width configurations and selecting optimal values that satisfy accuracy requirements while minimizing resource usage, the system resolves the contradiction between using fewer bits for efficiency and maintaining sufficient precision for accuracy
2Measurement precision
If floating point formats are used to maintain accuracy, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent changes the numerical representation format from floating point to fixed point with optimized bit widths. By carefully selecting appropriate bit width parameters for fixed point representation, the system achieves accuracy comparable to floating point while significantly reducing hardware complexity and resource requirements
Solution Approach 2:
The patent uses simplified fixed point arithmetic operations instead of complex floating point units. This substitution of computationally expensive floating point operations with simpler fixed point operations reduces hardware complexity while maintaining sufficient accuracy through optimized bit width selection
3Device complexity
If uniform quantisation is applied to all data types, then device complexity is reduced, but measurement precision deteriorates
Solution Approach 1:
The patent applies local quality by assigning different bit widths to different data types (activations, weights, biases) based on their specific requirements. Each data type receives a customized precision level tailored to its characteristics and the needs of the layers it feeds into, optimizing both accuracy and resource efficiency locally rather than applying a uniform quantisation scheme globally
Data Source
AI summary
One or more quantisation parameters are identified for transforming values to be processed by a Neural Network (NN) implemented in hardware. An output of a model of the NN is determined in response to training data, the model comprising quantisation blocks, each of which is configured to transform sets of values input to a layer of the NN to a respective fixed point number format defined by quantisation parameters prior to the model processing the sets of values in accordance with the layer. A cost metric of the NN is determined that is a combination of an error metric and an implementation metric representative of an implementation cost of the NN based on the quantisation parameters. The implementation metric is dependent on a first contribution representative of an implementation cost of an output from a layer, and a second contribution representative of an implementation cost of an output from a preceding layer. A derivative of the cost metric is back-propagated to at least one of the quantisation parameters to generate a gradient of the cost metric for at least one of the quantisation parameters, and the at least one quantisation parameter is adjusted based on the gradient.