Sparse Tensor Accelerator Architecture for Mixed Dense Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning (ML) processing engines face inefficiencies in handling a mix of sparse and dense tensors, leading to increased processing time, power consumption, and storage requirements, with challenges in converting between tensor forms and optimizing operations for various ML frameworks like TENSORFLOW.
Innovation Solution
The implementation of a processor with sparse/dense tensor processing circuits that include adapter input and output circuits, location processors, and arithmetic logic units (ALUs) to efficiently process and convert between sparse and dense tensor forms, minimizing computations and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sparse tensor compression is applied to reduce data footprint, then storage size and access bandwidth are reduced, but processing complexity increases due to the need for specialized sparse tensor processing circuits
Solution Approach 1:
The processing engine is segmented into specialized circuits: adapter input circuit for sparse tensor decomposition, adapter output circuit for recompression, and ALU array for computation. This segmentation allows each component to handle specific tasks efficiently, managing the complexity of sparse tensor processing through modular architecture.
Solution Approach 2:
The adapter input circuit acts as an intermediary that converts compressed sparse tensor format into a form suitable for ALU processing, while the adapter output circuit serves as another intermediary that converts processed results back to sparse format. These intermediary circuits enable the use of standard ALUs for sparse tensor operations without requiring complete redesign of the computing architecture.
2Ease of operation
If dense tensor operations are used for all tensor types, then processing is simpler and more uniform, but processing time and power consumption increase due to unnecessary computations on zero-value elements
Solution Approach 1:
The processing engine dynamically adapts its operation mode based on the input tensor type. The adapter input circuit detects whether input tensors are sparse or dense and configures the processing pipeline accordingly. For sparse tensors, the system uses compressed format processing with location tracking; for dense tensors, it uses standard uniform processing. This dynamic adaptation allows the system to optimize for speed when processing sparse data while maintaining simplicity for dense data.
3Device complexity
If conversion between sparse and dense tensor forms is performed externally, then the processing engine architecture remains simple, but overall system processing time increases due to data transfer overhead
Solution Approach 1:
The conversion functions are merged into the processing engine itself. The adapter input circuit combines sparse tensor decomposition with input loading, while the adapter output circuit combines recompression with output storage. This merging eliminates separate conversion steps and data transfers, allowing conversion operations to be performed in-place within the engine's memory hierarchy, thereby reducing overall processing time.
Solution Approach 2:
The adapter input circuit performs preliminary decomposition of sparse tensors into location-value pairs before the data enters the main processing pipeline. This preliminary action prepares the data in a format that enables efficient processing without requiring subsequent conversion steps, reducing total processing time by performing conversion work upfront.
4Use of energy by moving object
If sparse tensor processing is implemented to minimize element computations, then power consumption is reduced, but the number of processing units required increases to maintain calculation speed
Solution Approach 1:
The ALU array is designed to handle both sparse and dense tensor operations universally. The same ALUs that perform standard arithmetic operations also process sparse tensor data when configured with the appropriate input format from the adapter circuit. This multi-functionality allows the system to maintain high calculation speed with fewer specialized processing units, as the general-purpose ALUs can be dynamically configured for different data formats.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
In an example, a processor for machine learning calculations is described. An adapter input circuit is operable to receive an input tensor. The adapter input circuit includes channels. A first channel of the channels is operable to process samples of the input tensor to generate pre-processed samples and to obtain locations of the samples. A location processor, coupled to the first channel, is operable to determine output locations in response to the locations. An arithmetic logic unit (ALU), coupled to the channels, is operable to calculate output samples from the pre-processed samples. An adapter output circuit, coupled to the location processor and the ALU, operable to process the output locations and the output samples to generate an output tensor.