Sparse Memory Hierarchy for Dual-Sparsity Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training technologies do not effectively utilize operand sparsity to maintain accuracy and reduce hardware overhead, particularly in General Matrix Multiply operations, as they only support weight sparsity and not dual-sparsity arrangements for both activation and weight tensors.
Innovation Solution
A memory system that includes a decompressor unit, a buffer unit, and a neural processing unit configured to handle dual-sparsity arrangements for both activation and weight tensors, allowing for structured or random sparsity densities, thereby enabling efficient computation and reducing memory traffic and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If weight sparsity is used to accelerate NPU operations, then computation speed is improved, but hardware overhead is increased and accuracy is degraded
Solution Approach 1:
The patent segments the sparsity handling into two distinct components: weight sparsity (N:M structured sparsity) and activation sparsity (random or structured). By separating these concerns and providing dedicated hardware support for each type, the system achieves acceleration without proportionally increasing hardware overhead. The weight buffer and activation buffer are separately optimized for their respective sparsity patterns.
Solution Approach 2:
The patent applies different sparsity handling strategies to different parts of the neural network data flow. Weight tensors use structured N:M sparsity with dedicated buffer support, while activation tensors can use random or structured sparsity with separate buffer optimization. This localized optimization allows each component to be tuned independently for its specific requirements.
2Quantity of substance
If weight sparsity is used to reduce memory usage, then storage size is reduced, but accuracy is degraded
Solution Approach 1:
The patent creates a composite sparsity approach by combining structured weight sparsity (N:M) with activation sparsity (random or structured). This composite strategy maintains accuracy by preserving the structured nature of weight connections while allowing flexibility in activation representation. The dual-sparsity buffers work together to maintain the integrity of the neural network computation.
3Loss of energy
If dual-sparsity arrangements are implemented for both activation and weight tensors, then memory traffic is reduced, but device complexity is increased
Solution Approach 1:
The patent designs the dual-sparsity buffer system to handle multiple sparsity patterns (structured N:M for weights, random or structured for activations) using a unified architecture. The weight buffer and activation buffer can be configured to handle different sparsity types, providing universal support for various neural network optimizations without requiring separate dedicated hardware for each sparsity pattern.
4Speed
If existing weight sparsity technology is used, then computation is accelerated, but versatility is limited to weight tensors only
Solution Approach 1:
The patent introduces dynamic configurability to the buffer system, allowing the sparsity pattern and density to be adjusted based on the specific neural network layer and tensor type being processed. The weight buffer can be configured for structured N:M sparsity when processing weights, while the activation buffer can be configured for random or structured sparsity when processing activations, providing adaptability across different computational scenarios.
Data Source
AI summary
A memory system and a method are disclosed for training a neural network model. A decompressor unit decompresses an activation tensor to a first predetermined sparsity density based on the activation tensor being compressed, and decompresses an weight tensor to a second predetermined sparsity density based on the weight tensor being compressed. A buffer unit receives the activation tensor at the first predetermined sparsity density and the weight tensor at the second predetermined sparsity density. A neural processing unit receives the activation tensor and the weight tensor from the buffer unit and computes a result for the activation tensor and the weight tensor based on first predetermined sparsity density of the activation tensor and based on the second predetermined sparsity density of the weight tensor.