Int9 Quantization Hardware for ALU Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware-based machine learning systems face complexity and latency issues when performing arithmetic logic unit (ALU) calculations on a mixture of quantized data formats, requiring format type tracking and inefficient rescaling of integer values.
Innovation Solution
A programmable hardware architecture that converts data into a uniform int9 format, eliminating the need for format type tracking and enabling symmetric quantization, thereby simplifying ALU calculations and reducing processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a mixture of quantized format types is used for ALU calculations, then the system can process various data formats, but the complexity and latency for ALU calculation increases due to format type tracking
Solution Approach 1:
The patent applies homogeneity by converting all quantized data to a uniform int9 format before ALU calculations. This eliminates the need to track multiple format types while maintaining the ability to process various original data formats. The conversion process ensures all data enters the ALU in the same format, simplifying the calculation pipeline.
Solution Approach 2:
The patent introduces an intermediary conversion step that transforms diverse quantized formats into a standardized int9 format before ALU processing. This intermediary layer handles the format conversion, allowing the ALU to operate on homogeneous data without needing to track or handle format diversity directly.
2Reliability
If format type tracking is implemented for mixture of quantized formats, then correct calculations can be performed, but processing latency increases
Solution Approach 1:
The patent performs format conversion as a preliminary action before data enters the ALU. By converting all quantized data to int9 format in advance, the system eliminates the need for format tracking during calculation, thereby reducing latency while maintaining accuracy. The conversion happens once before processing, not repeatedly during operations.
Solution Approach 2:
The patent extracts the format conversion operation from the ALU calculation pipeline and performs it separately before data enters the ALU. This separation removes the format tracking burden from the critical calculation path, reducing latency while preserving calculation accuracy through dedicated conversion logic.
3Measurement precision
If mathematical division is used for rescaling integer values, then accurate rescaling can be achieved, but the operation is time consuming and inefficient
Solution Approach 1:
The patent substitutes the mechanical division operation with a multiplication-based rescaling approach. Instead of performing time-consuming division operations to rescale integer values, the system uses multiplication with pre-computed scaling factors, which is significantly faster and more efficient in hardware while maintaining the required precision.
Solution Approach 2:
The patent changes the rescaling parameter from division to multiplication by inverting the scaling factor. This parameter transformation allows the same rescaling function to be achieved through a more efficient operation (multiplication) that is faster in hardware implementations while preserving the accuracy of the rescaling process.
Data Source
AI summary
A method of converting a data stored in a memory from a first format to a second format is disclosed. The method includes extending a number of bits in the data stored in a double data rate (DDR) memory by one bit to form an extended data. The method further includes determining whether the data stored in the DDR is signed or unsigned data. Moreover, responsive to determining that the data is signed, a sign value is added to the most significant bit of the extended data and the data is copied to lower order bits of the extended data. Responsive to determining that the data is unsigned, the data is copied to lower order bits of the extended data and the most significant bit is set to an unsigned value, e.g., zero. The extended data is stored in an on-chip memory (OCM) of a processing tile of a machine learning computer array.