Tensor Accelerator Architecture for Sparse Activation Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity of artificial neural networks (ANNs) leads to bottlenecks in training and inference due to massive linear tensor operations, which are resource-intensive and power-consuming when solely executed on central processing units (CPUs).
Innovation Solution
An accelerator is designed to efficiently perform operations on tensors by combining sparse tensors into complementary dense tensors, using a routing circuit to group and separate active values, and incorporating adder trees and activation circuits to optimize multiplications and accumulations, thereby reducing the number of operations required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If sparse tensors are processed using conventional CPU-based systems, then flexibility and ease of instantiation are maintained, but computational resources and power consumption increase significantly
Solution Approach 1:
The system segments the tensor processing workload by separating sparse tensor operations from dense tensor operations. Sparse tensors are processed using specialized hardware circuits (multiply circuits and adder trees) that exploit the sparsity pattern, while dense tensors use conventional processing units. This segmentation allows the system to optimize for energy efficiency in sparse operations while maintaining flexibility through software-based dense operations.
Solution Approach 2:
The patent introduces an intermediary tensor reordering circuit that transforms sparse tensors into a complementary dense format before processing. This intermediary representation allows the system to use efficient dense matrix multiplication hardware while preserving the computational benefits of sparsity. The reordering circuit acts as a mediator between the sparse input format and the dense processing architecture.
2Productivity
If sparse tensors are processed using specialized hardware circuits, then computational speed and energy efficiency improve, but device complexity increases
Solution Approach 1:
The multiply circuits and adder trees are designed to handle both sparse and dense tensor operations. By making these circuits universal, the system achieves high computational speed for sparse operations without requiring entirely separate specialized hardware paths. The same hardware infrastructure serves multiple functions, reducing overall device complexity while maintaining productivity benefits.
Solution Approach 2:
The system changes the representation parameters of sparse tensors by reordering elements into a complementary dense format. This parameter transformation allows conventional dense matrix multiplication hardware to process sparse data efficiently. By changing how the data is organized rather than creating entirely new processing circuits, the system improves speed without proportionally increasing hardware complexity.
3Ease of operation
If conventional CPU systems are used for tensor operations, then ease of operation is maintained, but the number of operations and computational resources required increase
Solution Approach 1:
The tensor reordering circuit automatically performs the transformation from sparse to complementary dense format without requiring manual intervention. The system self-manages the complex data reorganization task, maintaining ease of operation for users while dramatically improving computational efficiency. The hardware automatically identifies and processes the sparsity patterns, freeing users from complexity.
Data Source
AI summary
A hardware accelerator that is efficient at performing computations related to tensors. The hardware accelerator may store a complementary dense process tensor that is combined from a plurality of sparse process tensors. The plurality of sparse process tensors have non-overlapping locations of active values. The hardware accelerator may perform elementwise operations between the complementary dense process tensor and an activation tensor to generate a product tensor. The hardware accelerator may re-arrange the product tensor based on a permutation logic to separate the products into groups. Each group corresponds to one of the sparse process tensors. Each group may be accumulated separately to generate a plurality of output values. The output values may be selected in an activation selection. The activation selection may be a dense activation or a sparse activation such as k winner activation that set non-winners to zeros.