Sparse Result Compression in AI Operation Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The bottleneck of input/output (I/O) bandwidth in AI operation accelerators for convolutional neural networks hinders efficient computation due to the large amount of weight and computation result data that needs to be transferred between the accelerator and external memory, affecting overall performance.
Innovation Solution
Incorporating a compression module and decompression module into the operation accelerator to reduce the data amount transferred by compressing and decompressing computation results, managed by a controller that determines when to perform these operations based on the sparse rate of input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weight data and computation results are transferred between external memory and operation accelerator, then computation can be performed, but I/O bandwidth becomes a bottleneck
Solution Approach 1:
The patent extracts only the essential information from computation results by identifying sparse non-zero elements and their positions, rather than transferring complete computation results. This extraction approach reduces data transfer volume while preserving necessary computational information for subsequent operations.
Solution Approach 2:
Instead of transferring complete data and filtering later, the patent inverts the approach by identifying and transferring only the necessary sparse elements directly. The compression module inverts the traditional data flow by selecting minimal essential data for transmission between memory and accelerator.
2Quantity of substance
If compression module is added to reduce data transfer, then I/O bandwidth is reduced, but device complexity increases
Solution Approach 1:
The compression module is designed to perform multiple functions: identifying sparse elements, encoding their positions and values, and managing data format conversion. This multi-functional design reduces the need for separate dedicated components for each compression task, thereby limiting the increase in device complexity.
3Quantity of substance
If compression is performed on computation results, then data amount is reduced, but power consumption increases
Solution Approach 1:
The patent applies partial compression by identifying and compressing only the sparse non-zero elements rather than compressing entire computation results. This partial action approach reduces power consumption compared to full compression while still achieving significant data reduction for sparse data patterns typical in neural network computations.
Data Source
AI summary
The present disclosure provides example operation accelerators and compression methods. One example operation accelerator includes a storage configured to store first input data, weight data, and a control instruction, and an operation circuit connected to the storage and configured to perform matrix multiplication on the first input data and the weight data, to obtain a computation result. The operation accelerator further includes a compression module configured to compress the computation result to obtain compressed data, as well as a controller connected to the storage and configured to obtain the control instruction from the storage, and when the control instruction includes instructions to compress the computation result, control the compression module to compress the computation result to obtain the compressed data. The operation accelerator further includes a direct memory access controller connected to the compression module and configured to store the compressed data in a memory outside the operation accelerator.