Enhanced Block Floating Point Format for Neural Network Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Block Floating Point (BFP) representation is inadequate for applications with large ranges of values, such as convolutional neural networks, as it sets small numbers to zero, leading to inaccurate results due to right-shifting of significands.
Innovation Solution
The Enhanced Block Floating Point (EBFP) format represents small numbers as the difference between the exponent and a shared exponent, using a tag bit to indicate whether the number represents a shifted significand or the exponent difference, allowing for more accurate representation of numbers with varying exponents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If conventional Block Floating Point representation is used to reduce computing resources, then resource usage decreases, but small numbers are set to zero causing loss of information
Solution Approach 1:
The significand storage is segmented into two distinct parts: a primary significand field for standard BFP representation and an additional field for storing exponent differences. This segmentation allows the system to preserve both large and small numbers within the same block without requiring full floating-point precision for each number, thus reducing computing resources while maintaining information about small values.
Solution Approach 2:
An intermediary mechanism is introduced where exponent differences are stored separately from the main significand. This intermediary storage allows the system to recover and use small number information that would otherwise be lost in conventional BFP, while still benefiting from the resource efficiency of block floating-point representation.
2Device complexity
If significands are right-shifted to represent a block of FP numbers by a shared exponent, then device complexity decreases, but accuracy deteriorates as small numbers become zero
Solution Approach 1:
The storage structure is segmented to include both the right-shifted significand and an additional component for exponent differences. This segmentation maintains the simplicity of BFP arithmetic while preserving accuracy information that would otherwise be lost during the right-shifting process.
Solution Approach 2:
The system changes the parameters of representation by storing not just the significand but also metadata about the exponent differences. This parameter change allows the system to maintain low device complexity while improving measurement precision for small numbers within the block.
3Quantity of substance
If a shared exponent is used for a block of FP numbers, then the number of bits required decreases, but the range of representable numbers is limited
Solution Approach 1:
The bit representation is segmented into multiple components: the shared exponent, the significand, and an additional field for exponent differences. This segmentation allows the system to use fewer total bits while expanding the effective range of representable numbers by capturing both the dominant large values and the subordinate small values within the block.
Solution Approach 2:
The system adds another dimension to the representation by storing exponent differences separately. This dimensional addition allows the block floating-point format to represent a wider range of numbers without increasing the total bit count significantly, as the exponent difference field efficiently encodes variations from the shared exponent.
Data Source
AI summary
In a data processor, an input value having a sign, an exponent and a significand is encoded by determining an exponent difference between a base exponent and the exponent. When the exponent difference is not less than a first threshold, only the exponent difference, or a designated value, is encoded to a payload of the output value and one or more tag bits of the output value are set to a first value. When the exponent difference is less than the first threshold, the significand and exponent difference are encoded to the payload of an output value and, optionally, the one or more tag bits of the output value. A sign bit in the output value is set corresponding to the sign of the input value, and the output value is stored.