Mixed-Precision MAC Tree for LLM Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative large language models require high precision operations that are resource-intensive and difficult to parallelize, leading to challenges in maximizing memory bandwidth usage and reducing hardware resource overhead.
Innovation Solution
A MAC tree-based operator is developed, comprising multiple floating-point multipliers connected in parallel, converters for precision conversion, a fixed-point adder tree, and an accumulator, which processes data from external memory, supporting mixed-precision operations to optimize memory bandwidth and reduce resource overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point operators are used to support generative large language model operations without loss of accuracy, then precision is maintained, but hardware resource overhead and logic complexity increase significantly
Solution Approach 1:
The patent segments the computational precision requirements into different parts: high-precision floating-point operations for critical calculations and low-precision fixed-point operations for less critical calculations. This segmentation allows the system to maintain necessary accuracy while reducing overall hardware complexity by not requiring all operations to use full floating-point precision.
Solution Approach 2:
The patent applies local quality by using different precision levels in different parts of the computational system. Specifically, floating-point precision is applied where accuracy is critical, while fixed-point precision is used where it suffices, optimizing the balance between precision and resource usage in different locations of the hardware architecture.
2Measurement precision
If more data is retrieved from memory at once to support larger model parameters, then model precision and capability improve, but memory bandwidth requirements increase
Solution Approach 1:
The patent changes the precision parameter of data representation from uniform floating-point to mixed-precision formats. By converting some data to lower-precision fixed-point representations, the system reduces the bandwidth required for memory operations while maintaining sufficient precision for the computational tasks, thus resolving the contradiction between model precision and memory bandwidth usage.
3Productivity
If the number of FP multipliers is increased to maximize memory bandwidth usage, then computational throughput improves, but hardware resource overhead increases
Solution Approach 1:
The patent segments the multiplier operations into floating-point multipliers for high-precision requirements and fixed-point multipliers for lower-precision requirements. This segmentation enables the system to achieve high computational throughput by utilizing multiple simpler fixed-point multipliers instead of fewer complex floating-point multipliers, thereby improving productivity while controlling hardware resource overhead.
Solution Approach 2:
The patent uses copying by creating multiple instances of simpler fixed-point multiplier units to replace or supplement more complex floating-point multiplier units. This approach achieves equivalent or superior computational throughput using less hardware resource overhead, as the simpler units can be replicated more efficiently.
Data Source
AI summary
Provided is a mixed-precision multiply-and-accumulation (MAC) tree structure to maximize memory bandwidth usage for computational acceleration of a generative large language model. A MAC tree-based operator may include a plurality of floating-point (FP) multipliers connected in parallel and configured to process a multiplication operation on data delivered from an external memory; a plurality of first converters configured to convert output of each of the plurality of FP multipliers from floating point to fixed point; a fixed-point (FXP) adder tree connected to the plurality of first converters and configured to process summation of multiplication results of the plurality of FP multipliers; an FXP accumulator configured to accumulate output of the FXP adder tree; and a second converter configured to convert output of the FXP accumulator from the fixed point to the floating point.