Block Floating-Point Mantissa Bit-Width Adjustment for Neural Network Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for processing artificial neural networks with large input data are inefficient, as they struggle to effectively extract information due to high computational requirements and inaccuracies at low precision levels, especially in sequence transduction models.
Innovation Solution
A processor-implemented method that adjusts the bit-width of the mantissa in floating-point data based on the exponent size, comparing it to a threshold determined by the data distribution and allowable error range, to optimize operations and reduce computational complexity while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional fixed-point or integer quantization methods are used, then hardware implementation is simplified, but inference accuracy deteriorates at very low word sizes due to inadequate dynamic range
Solution Approach 1:
The patent implements block floating-point representation where the exponent is shared across a block of mantissas, allowing dynamic adjustment of the dynamic range. This enables the system to adapt to varying data distributions while maintaining hardware efficiency, resolving the contradiction between simplified hardware and adequate dynamic range for low precision.
Solution Approach 2:
The patent changes the numerical representation parameters by introducing block floating-point format with shared exponents and optimized mantissa bit-widths. This parameter change allows the system to achieve both hardware simplicity and sufficient dynamic range by fundamentally altering how numbers are represented and processed.
2Measurement precision
If uniform floating-point representation is used, then dynamic range is maximized, but computational efficiency deteriorates due to variable precision requirements
Solution Approach 1:
The patent segments the floating-point representation into blocks where multiple values share a common exponent. This segmentation allows uniform treatment of exponent management while reducing the overall computational burden by processing groups of values together rather than individually, thus improving computational efficiency while maintaining dynamic range.
Solution Approach 2:
The shared exponent structure provides universal handling for multiple mantissas within a block, allowing a single exponent management mechanism to serve multiple values. This multi-functional approach improves computational efficiency by eliminating redundant exponent operations while maintaining the dynamic range benefits of floating-point representation.
3Measurement precision
If higher precision floating-point is used, then inference accuracy is improved, but energy consumption and computational complexity increase
Solution Approach 1:
The patent optimizes the precision parameters by using block floating-point with shared exponents and carefully selected mantissa bit-widths. This parameter optimization maintains inference accuracy by preserving sufficient precision where needed while reducing the overall bit-width and computational complexity, thereby lowering energy consumption.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A processor-implemented method includes: receiving a plurality of pieces of input data expressed as floating point; adjusting a bit-width of mantissa by performing masking on the mantissa of each piece of the input data based on a size of an exponent of each piece of the input data; and performing an operation between the input data with the adjusted bit-width.