Floating Point Multiply-Accumulate Unit with Speculative Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face significant computation costs due to high demands for multiply-accumulate operations and weight operand storage, particularly in energy-constrained mobile and embedded systems, where existing FPMAC units are inefficient and fail to optimize half-precision floating-point operations, leading to performance and energy bottlenecks.
Innovation Solution
The proposed solution involves a DNN accelerator with a data storing module that splits operands into two portions, a concatenating module that sequences these portions for efficient processing, and a processing element (PE) array with flexible FPMAC units that perform various compute patterns, including vector-vector, matrix-vector, and matrix-matrix operations, along with speculative alignment and separated critical paths for alignment and normalization, enhancing space and energy efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional FPMAC units are used for deep neural network inference, then floating-point computation can be performed, but energy consumption is excessive and processing speed is slow in mobile and embedded systems
Solution Approach 1:
The FPMAC unit is segmented into separate alignment logic and critical path logic. The alignment logic handles exponent comparison and mantissa alignment independently, while the critical path logic focuses on multiplication and accumulation. This segmentation allows parallel processing of alignment and computation, improving throughput while reducing per-operation energy consumption by avoiding redundant operations in the critical path.
Solution Approach 2:
Exponent alignment is performed as a preliminary action before the critical multiplication and accumulation path. The alignment logic prepares the operands by comparing exponents and shifting mantissas in advance, so that when data enters the critical path, the multiplication can proceed immediately without waiting for alignment completion. This preliminary action increases processing speed while the energy cost is distributed across non-critical stages.
2Measurement precision
If high-precision floating-point operations are performed to maintain accuracy, then computation accuracy is improved, but computation cost and resource requirements increase significantly
Solution Approach 1:
The FPMAC unit supports dynamic parameter changes in floating-point precision (e.g., switching between FP32, FP16, BF16 formats). By allowing the precision parameters to be adjusted based on application requirements, the system can maintain high accuracy when needed while reducing computational cost for less demanding operations. The hardware is designed to handle various precision levels efficiently without requiring separate dedicated units for each precision level.
3Productivity
If separate alignment and critical path logic are implemented, then processing throughput is improved, but device area increases
Solution Approach 1:
The alignment logic and critical path logic are merged into a single integrated FPMAC unit structure. The alignment logic feeds directly into the critical path without requiring separate physical units, allowing data to flow continuously through both stages. This merging achieves high throughput by eliminating inter-unit data transfer delays while controlling area growth through shared resource utilization and efficient spatial arrangement of logic elements within the unified structure.
Data Source
AI summary
A FPMAC operation has two operands: an input operand and a weight operand. The operands may have a format of FP16, BF16, or INT8. Each operand is split into two portions. The two portions are stored in separate storage units. Then operands are transferred to register files of a PE, with each register file storing bits of an operand sequentially. The PE performs the FPMAC operation based on the operands. The PE may include an FPMAC unit configured to compute an individual partial sum of the PE. The PE may also include an FP adder to accumulate the individual partial sum with other data, such as an output from another PE or an output form another PE array. The FP adder may be fused with the FPMAC unit in a single circuit that can do speculative alignment and has separate critical paths for alignment and normalization.