Systolic Array Output Rounding for Mixed-Precision Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systolic arrays in artificial neural networks suffer from loss of statistical information during output rounding due to conventional rounding methods, which negatively impact the accuracy of computations, especially in latency-sensitive inference phases.
Innovation Solution
Implement stochastic rounding in systolic arrays using pseudo-random number generators, such as linear feedback shift registers, to maintain statistical information by randomly rounding outputs based on the distance between unrounded and rounded values, ensuring a low expected error.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional rounding methods are used in systolic arrays, then the device complexity is reduced, but statistical information is lost during output rounding
Solution Approach 1:
The patent changes the rounding parameter from deterministic (conventional rounding) to stochastic (probability-based). By introducing a probability parameter p that determines whether to round up or down, the system preserves statistical information while maintaining compatibility with standard rounding hardware. This parameter change transforms the rounding behavior to maintain expected value equality with the original unrounded value.
Solution Approach 2:
The patent introduces an intermediary rounding mechanism that sits between the systolic array computation and the final output. This intermediary layer implements the stochastic rounding logic using simple components (comparators, multiplexers, and probability generators) that mediate the rounding process to preserve statistical properties without requiring complex hardware modifications throughout the entire systolic array.
2Productivity
If lower-bit-length data formats are used, then the processing speed increases, but the computation accuracy decreases
Solution Approach 1:
The patent changes the data format parameter from high-bit-length to lower-bit-length formats (such as FP16 or INT8), which increases processing speed by reducing the computational burden. To compensate for the potential accuracy loss, the patent applies stochastic rounding that preserves statistical information, ensuring that the expected value of rounded results matches the original high-precision results.
Solution Approach 2:
The patent implements a feedback mechanism where the rounding probability p is dynamically adjusted based on the distance between the unrounded value and the rounded value. This feedback ensures that the stochastic rounding process continuously adapts to maintain accuracy while benefiting from the speed advantages of lower-bit-length formats.
3Measurement precision
If stochastic rounding is implemented using pseudo-random number generators, then the accuracy of computations is maintained, but the device complexity increases
Solution Approach 1:
The patent introduces an intermediary rounding mechanism that sits between the systolic array computation and the final output. This intermediary layer implements the stochastic rounding logic using simple components (comparators, multiplexers, and probability generators) that mediate the rounding process to preserve statistical properties without requiring complex hardware modifications throughout the entire systolic array.
Solution Approach 2:
The patent uses pseudo-random number generators that can be replicated across multiple processing elements in the systolic array. By copying the same LFSR design to each processing element, the system achieves parallel stochastic rounding with minimal additional complexity, as each element independently generates its own pseudo-random sequence for rounding decisions.
Data Source
AI summary
Systems and methods are provided to round the numbers produced by a systolic array. A rounder can obtain a number from the systolic array and identify a data type conversion associated with the number. The data type conversion may indicate a first bit-length and a second bit-length of the number. The rounder can select a random number generator for rounding the number based on the data type conversion. The bit-length of the random number generator may be equal to a difference in bit-length between the first bit-length and the second bit-length. The rounder can perform a rounding operation using a random number generated by the random number generator.