Sparse Tensor Compression for Deep Learning Acceleration Chips
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning acceleration chips face inefficiencies due to physical restrictions on data movement and high power consumption, with operation efficiency limited by the Amdahl law at 10 TOPs/W, necessitating a breakthrough to enhance performance.
Innovation Solution
Implementing a data compression method that involves matrix decomposition to convert filter coefficient tensor matrices into sparse tensor matrices and orthonormal transformation matrices, which are then stored and used for convolution operations, reducing data movement and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is moved frequently between memory and processing units, then operation speed can be improved, but power consumption increases and physical restrictions are exceeded
Solution Approach 1:
The patent applies preliminary action by pre-processing filter coefficient tensors into sparse tensors and storing them in on-chip memory before processing. This allows the processing unit to perform multiple operations using the same pre-processed data without repeated memory accesses, thereby reducing power consumption while maintaining high operation speed.
Solution Approach 2:
The patent implements nesting by organizing memory structures hierarchically - sparse tensors are stored in on-chip memory (higher level) which is nested within the processing unit, while original tensors remain in external memory (lower level). This nested architecture allows fast access to frequently used sparse data without repeatedly accessing external memory, resolving the contradiction between speed and power consumption.
2Ease of operation
If the shared bus is used for data movement, then data exchange between memory and processing units is enabled, but the bus becomes occupied and operation efficiency decreases
Solution Approach 1:
The patent segments the data storage architecture into multiple independent memory spaces: on-chip memory for sparse tensors and external memory for original tensors. This segmentation eliminates the need for a single shared bus, allowing parallel data access and processing, thereby improving operation efficiency while maintaining data exchange capability.
Solution Approach 2:
The patent introduces on-chip memory as an intermediary between external memory and the processing unit. This intermediary stores frequently accessed sparse tensors locally, reducing the need for repeated transactions over the shared bus and improving overall operation efficiency without compromising data exchange capabilities.
3Quantity of substance
If external memory is accessed for data operations, then high capacity storage is available, but primary power loss occurs due to access operations
Solution Approach 1:
The patent extracts the most frequently accessed data (sparse tensors) from external memory and places them in on-chip memory. This extraction reduces the number of power-consuming access operations to external memory, thereby reducing power loss while maintaining access to high-capacity storage when needed.
Solution Approach 2:
The patent applies local quality by providing different memory characteristics in different locations: on-chip memory offers fast, low-power access for frequently used sparse tensors, while external memory provides high-capacity storage for less frequently accessed original tensors. This localized optimization reduces overall power loss while maintaining total storage capacity.
Data Source
AI summary
A data compression method, a data compression system and an operation method of a deep learning acceleration chip are provided. The data compression method includes the following steps. A filter coefficient tensor matrix of a deep learning model is obtained. A matrix decomposition procedure is performed according to the filter coefficient tensor matrix to obtain a sparse tensor matrix and a transformation matrix, which is an orthonormal matrix. The product of the transformation matrix and the filter coefficient tensor matrix is the sparse tensor matrix. The sparse tensor matrix is compressed. The sparse tensor matrix and the transformation matrix, or the sparse tensor matrix and a restoration matrix, are stored in a memory. A convolution operation result is obtained by the deep learning acceleration chip using the sparse tensor matrix. The convolution operation result is restored by the deep learning acceleration chip using the restoration matrix.