INT8 Winograd Convolution Calibration for Inference Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Coppersmith-Winograd algorithm is limited to 32-bit floating point operations and has not been successfully applied in 8-bit integer (INT8) settings for deep neural networks, leading to challenges in reducing computational complexity and maintaining accuracy in convolution operations.
Innovation Solution
A calibration approach is used to determine scale factors for 8-bit integer data types, allowing the Coppersmith-Winograd algorithm to be applied in INT8 settings without significant accuracy loss by transforming and quantizing FP32 tensors to INT8 tensors using quantization and dequantization functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the Coppersmith-Winograd algorithm is applied to 8-bit integer operations, then computational complexity is reduced and hardware performance is improved, but accuracy is compromised compared to 32-bit floating point implementations
Solution Approach 1:
The patent applies parameter changes by transforming the data type from 32-bit floating point to 8-bit integer, adjusting the precision parameter to enable INT8 operations. This allows the Coppersmith-Winograd algorithm to operate with reduced computational complexity while maintaining hardware efficiency, directly addressing the contradiction between productivity and measurement precision.
Solution Approach 2:
The patent introduces calibration data as an intermediary element that bridges the gap between FP32 training and INT8 inference. By using calibration data to compute scale factors and transformation matrices, the system maintains accuracy during the precision reduction process, enabling accurate INT8 convolution operations without significant loss compared to FP32 implementations.
2Quantity of substance
If 8-bit integer data types are used instead of 32-bit floating point, then memory requirements are reduced and power consumption is lowered, but computational accuracy deteriorates
Solution Approach 1:
The patent changes the data type parameter from FP32 to INT8, reducing the bit width from 32 to 8. This parameter change directly reduces memory requirements and power consumption while the calibration process ensures that accuracy is maintained through proper scaling and transformation of the data.
Solution Approach 2:
The patent performs preliminary calibration actions before the actual inference to compute scale factors and transformation parameters. This preliminary action with calibration data ensures that when INT8 operations are executed, the accuracy is preserved by pre-adjusting the data distribution and range to match the reduced precision requirements.
3Reliability
If the Coppersmith-Winograd algorithm is limited to 32-bit floating point operations, then algorithm correctness is maintained, but computational efficiency and hardware acceleration are restricted
Solution Approach 1:
The patent changes the operational parameter from FP32 to INT8, enabling the Coppersmith-Winograd algorithm to run on 8-bit integer hardware. This parameter change maintains algorithm correctness through proper calibration and transformation, while significantly improving computational efficiency and hardware acceleration capabilities.
Solution Approach 2:
The patent substitutes the mechanical system of 32-bit floating point arithmetic with 8-bit integer arithmetic. This substitution enables the use of more efficient integer-based hardware units, achieving faster convolution computation while maintaining correctness through calibration-based transformation that preserves the mathematical properties required for accurate inference.
Data Source
AI summary
Various embodiments are generally directed to convolutional neural networks (CNN). A calibration dataset and a pretrained CNN comprising 32-bit floating point weight values may be sampled to generate an input activation tensor and a weight tensor. A transformed input activation tensor may be generated by multiplying the input activation tensor and an input matrix to generate a transformed input activation tensor. A transformed weight tensor may be generated by multiplying the weight tensor and a weight matrix. A scale factor may be computed for each transformed tensor. An 8-bit CNN model including the scale factors may be generated.