Matrix Vector Unit Block-Floating Point for Neural Network Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing technologies face inefficiencies in floating-point operations due to the need for individual exponent representation for each value, leading to high bit requirements and configuration complexity, especially in deep learning tasks like LSTM networks, which are computationally intensive.
Innovation Solution
Implementing a block-floating point (BFP) representation that uses a shared exponent for subsets of vector data, allowing for a higher dynamic range and reducing the bit requirements, combined with a matrix vector unit (MVU) for parallel processing on FPGAs, which includes pipelined multifunction units (MFUs) for vector operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If individual exponent representation is used for each floating-point value, then measurement precision is maintained, but device complexity and bit requirements increase significantly
Solution Approach 1:
The patent segments the vector data into multiple blocks, where each block shares a common exponent. This segmentation approach maintains floating-point precision within each block while reducing the overall number of exponents needed, thereby decreasing device complexity and configuration overhead without sacrificing measurement precision.
Solution Approach 2:
The patent implements a universal exponent representation where a single exponent value serves multiple floating-point values within a block. This multi-functionality allows the same exponent to be reused across multiple data elements, reducing the bit requirements and configuration complexity while maintaining the precision needed for neural network computations.
2Device complexity
If block-floating point representation with shared exponent is used, then device complexity is reduced, but measurement precision may be compromised
Solution Approach 1:
By segmenting data into smaller blocks with shared exponents, the patent minimizes the precision loss that would occur with a single global exponent. Each block can be processed independently with its own shared exponent, maintaining better numerical precision while still achieving the complexity reduction benefits of block-floating-point representation.
Solution Approach 2:
The patent dynamically determines the shared exponent for each block based on the actual data range within that block. This dynamic adaptation ensures that each block uses an exponent that is optimally suited to its specific values, thereby maintaining measurement precision while benefiting from the reduced complexity of shared exponent representation.
3Measurement precision
If traditional floating-point processing is used, then measurement precision is maintained, but processing speed decreases due to sequential operations
Solution Approach 1:
The patent segments vector data into multiple blocks that can be processed in parallel. Each block with its shared exponent can be handled independently by different processing units simultaneously, dramatically increasing processing speed while maintaining floating-point precision through the use of appropriate exponent values for each block.
Solution Approach 2:
The patent processes data in blocks rather than requiring complete sequential processing of all values. This partial action approach allows subsets of the data to be processed concurrently, improving overall processing speed while the block-floating-point representation ensures that precision requirements are met for each processed subset.
4Measurement precision
If more bits are allocated for exponent representation, then measurement precision is improved, but resource overhead increases
Solution Approach 1:
By segmenting data into blocks that share exponents, the patent reduces the total number of exponents needed across the entire dataset. This segmentation allows for more efficient bit allocation where precision is concentrated where needed within each block rather than requiring high precision for every individual value, thereby reducing overall bit requirements while maintaining necessary measurement precision.
Solution Approach 2:
The shared exponent mechanism allows a single exponent representation to serve multiple data values, reducing the total quantity of bits required for exponent storage. This universal representation maintains measurement precision for all values within the block while significantly reducing the overall bit requirements compared to individual exponent representations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processors and methods for neural network processing are provided. A method includes receiving vector data corresponding to a layer of a neural network model, where each of the vector data has a value comprising at least one exponent. The method further includes first processing a first subset of the vector data to determine a first shared exponent for representing values in the first subset of the vector data in a block-floating point format and second processing a second subset of the vector data to determine a second shared exponent for representing values in the second subset of the vector data in a block-floating point format in a manner that no vector data from the second subset of the vector data influences a determination of the first shared exponent and no vector data from the first subset of the vector data influences a determination of the second shared exponent.