Enhanced Block Floating Point Format for Neural Network Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Block Floating-Point (BFP) representation is not well-suited for applications with a large range of values, such as convolutional neural networks, as it sets small numbers to zero, leading to inaccurate results due to right-shifting of significands, and is limited by storage resources in data-intensive applications.
Innovation Solution
The Enhanced Block Floating Point (EBFP) format represents numbers using a shared exponent and a payload that includes either the encoded significand or the exponent difference, allowing for more accurate representation of small numbers and efficient storage by varying the allocation between exponent and significand bits, and includes mechanisms for rounding and special values to preserve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Block Floating-Point (BFP) representation is used to increase the range of representable numbers, then the range is increased compared to fixed-point, but small numbers are replaced by zero when significands are right-shifted too far, leading to loss of precision
Solution Approach 1:
The patent applies dynamics by making the exponent representation flexible rather than fixed. Each number in the block can have its own stored exponent value, allowing the system to adapt to the actual magnitude of each number dynamically. This resolves the contradiction by enabling small numbers to maintain their exponent information instead of being uniformly right-shifted to zero, thus preserving precision while maintaining the block-based efficient computation structure.
Solution Approach 2:
The patent changes the parameter representation by storing individual exponent differences for each number in the block rather than using a single shared exponent. This parameter change allows each number to be represented with its appropriate scale factor, preventing small numbers from being lost during the conversion process while still benefiting from the block floating-point format's efficiency in representing ranges of values.
2Measurement precision
If higher precision number representations are used to maintain accuracy, then computation accuracy is improved, but storage resources are exceeded in data-intensive applications
Solution Approach 1:
The patent applies partial action by storing only the necessary exponent information for each number rather than the complete floating-point representation. By storing just the exponent difference (a small number of bits) along with the significand, the system achieves high precision for computation while using significantly less storage space than full floating-point formats, thus resolving the contradiction between precision and storage consumption.
3Productivity
If conventional BFP uses a shared exponent for a block of numbers, then computation efficiency is improved, but the range representation is limited and small numbers are lost
Solution Approach 1:
The patent applies segmentation by dividing the exponent representation into individual exponent differences for each number in the block, rather than using a single shared exponent. This segmentation allows each number to be processed with its appropriate scale while still maintaining the block-based structure for efficient computation, thus resolving the contradiction between computation efficiency and precision by having the best of both approaches.
Data Source
AI summary
An apparatus and method of converting data into an Enhanced Block Floating Point (EBFP) format with a shared exponent is provided. The EBFP format enables data within a wide range of values to be stored using a reduced number of bits compared with conventional floating-point or fixed-point formats. The data to be converted may be in any other format, such as fixed-point, floating-point, block floating-point or EBFP.