Low-Precision Neural Network Conversion for Stable INT8 Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for neural network inference using reduced precision data formats face challenges such as numerical overflow, underflow, and require additional processing overhead due to dynamic quantization and fine-tuning during training.
Innovation Solution
A computer-implemented process that automates the conversion from higher precision data formats to lower precision formats by generating candidate conversions, measuring divergence using a histogram, and selecting the conversion with the least divergence for accurate inference, eliminating the need for re-training or fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If reduced precision data formats (float16, INT8) are used for neural network inference, then memory storage requirements and processing time are reduced, but numerical overflow and underflow occur due to narrower numerical range
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the precision format parameters based on the input data characteristics. The system analyzes the input data range and selectively applies different precision formats (FP32, FP16, INT8) to different portions of the neural network computation, thereby optimizing inference speed while maintaining numerical stability through adaptive parameter selection rather than fixed precision conversion
2Quantity of substance
If dynamic quantization is applied during training to convert to lower precision formats, then memory and processing requirements are reduced, but additional processing overhead is introduced
Solution Approach 1:
The patent implements preliminary action by performing precision format selection and data conversion during the training phase. The system pre-processes the training data to determine optimal precision formats for different layers and data types, storing these conversion parameters for reuse during inference. This eliminates the need for dynamic quantization during inference, reducing processing overhead while maintaining memory efficiency
3Reliability
If constraints are applied during training to prevent overflow and underflow in reduced precision formats, then numerical stability is improved, but the training process becomes more complex and requires fine-tuning
Solution Approach 1:
The patent introduces an intermediary mechanism in the form of a precision management layer that sits between the training algorithm and the reduced precision data formats. This intermediary automatically handles precision conversion, range checking, and format selection based on pre-computed statistics from the training data, thereby maintaining numerical stability without requiring complex constraints or manual fine-tuning of the training process
Data Source
AI summary
Aspects of the present invention are directed to computer-implemented techniques for performing data compression and conversion between data formats of varying degrees of precision, and more particularly for improving the inferencing (application) of artificial neural networks using a reduced precision (e.g., INT8) data format. Embodiments of the present invention generate candidate conversions of data output, then employ a relative measure of quality to identify the candidate conversion with the greatest accuracy (i.e., least divergence from the original higher precision values). The representation can be then be used during inference to perform computations on the resulting output data.