AI Operation Accelerator Compression for I/O Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high I/O bandwidth requirements for weight data import and computation result export in AI operation accelerators bottleneck their performance, particularly due to the large amount of data involved in convolutional neural network operations, which hampers overall efficiency.
Innovation Solution
Incorporating a compression module and a direct memory access controller into the operation accelerator to compress computation results and store them externally, while also including a decompression module to handle compressed data for subsequent computations, thereby reducing data transfer and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weight data is imported from external memory to on-chip buffer for Conv and FC operations, then computation can be performed, but I/O bandwidth is consumed and becomes a performance bottleneck
Solution Approach 1:
The patent applies preliminary action by pre-loading weight data into the on-chip buffer before computation is needed. The system predicts which weight data will be needed based on the computational graph and loads it in advance, reducing the need for frequent I/O operations during actual computation and thus alleviating the I/O bandwidth bottleneck while maintaining high computing performance.
2Speed
If computation results are stored in on-chip buffer, then fast access is enabled, but the buffer quickly becomes full requiring export to external memory, increasing I/O operations
Solution Approach 1:
The patent applies preliminary action by predicting which computation results will be needed by subsequent operations and prioritizing their storage in the on-chip buffer. This predictive approach ensures that frequently accessed results remain in fast memory longer, reducing the frequency and volume of exports to external memory, thereby maintaining high access speed while minimizing data export time.
3Quantity of substance
If compression module is added to reduce I/O bandwidth, then bandwidth usage decreases, but device complexity increases
Solution Approach 1:
The patent applies parameter changes by implementing a compression module that dynamically adjusts the compression level and algorithm based on the characteristics of the data being transferred. This allows the system to achieve significant I/O bandwidth reduction by transforming data parameters (compressing them) while the added complexity is managed through intelligent control that adapts to different computational scenarios.
Data Source
AI summary
The present disclosure discloses example operation accelerators and compression methods. One example operation accelerator performs operations, including storing, in a first buffer, first input data. In a second buffer, weight data can be stored. A computation result is obtained by performing matrix multiplication on the first input data and the weight data by an operation circuit connected to the input buffer and the weight buffer. The computation result is compressed by a compression module to obtain compressed data. The compressed data can be stored into a memory outside the operation accelerator by a direct memory access controller (DMAC) connected to the compression module.