Accelerator Cache Precision Conversion for Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The performance gap between the memory bandwidth of external memory and the computation performance of matrix multiplication computations in AI processing has not been adequately addressed, hindering the increase in speed of AI processing.
Innovation Solution
A computing system with a cache memory (LLC) is introduced between the accelerator chip and external memory, which reduces the accuracy of matrix values through conversion based on the range of exponent parts, allowing efficient use of memory bandwidth by selecting lower-precision representation formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision data is transferred from external memory to accelerator, then computation accuracy is improved, but memory bandwidth utilization is reduced and transfer time increases
Solution Approach 1:
The patent dynamically changes the precision parameter of matrix data based on the range of exponent parts. When the exponent range is large, the system converts data to lower-precision formats (FP8, BF16) to reduce transfer time and memory bandwidth consumption. When precision is critical, it uses higher-precision formats (FP32). This parameter adaptation resolves the contradiction by selecting the minimum necessary precision for each operation.
Solution Approach 2:
The patent implements dynamic precision conversion based on real-time analysis of exponent part ranges. The system continuously monitors the data characteristics and adjusts the precision level dynamically during computation. This dynamic approach allows the system to optimize the balance between accuracy and transfer efficiency for each specific computation task.
2Productivity
If data precision is reduced through conversion, then memory bandwidth utilization is improved, but computation accuracy deteriorates
Solution Approach 1:
The system changes the precision parameter based on the actual data characteristics (exponent range). By analyzing the exponent part range before conversion, the system selects the appropriate target precision level that maintains sufficient accuracy for the computation while maximizing memory bandwidth utilization. This data-driven parameter selection ensures accuracy is not unnecessarily reduced.
Solution Approach 2:
The patent applies partial precision reduction rather than complete reduction. It converts only the necessary portion of data to lower precision when the exponent range justifies it, and maintains higher precision when needed. This partial action approach prevents excessive precision loss while still achieving significant memory bandwidth improvements where appropriate.
3Speed
If cache memory is introduced between accelerator and external memory, then data access speed is improved, but device complexity increases
Solution Approach 1:
The patent introduces a cache memory as an intermediary component between the accelerator and external memory. This cache layer handles the precision conversion and data buffering functions, acting as a mediator that simplifies the overall system architecture. By centralizing the conversion logic in the cache, the system reduces the complexity burden on individual components while maintaining high data access speed.
Data Source
AI summary
A computing system includes: a plurality of accelerators that perform matrix multiplication computations; a cache memory that caches data of an external memory that saves a computation result by each of the plurality of accelerators; and a controller configured to: determine whether or not an access to the cache memory is congested; and in a case where it is determined that the access is congested, control the cache memory to perform conversion for reducing accuracy of a value of each component of a matrix on the matrix read from the external memory in response to an access from one accelerator that is one of the plurality of accelerators and is configured to perform the matrix multiplication computation of the matrix and another matrix saved in the external memory, and to transfer the matrix.