Hierarchical Weight Preprocessing for Sparse Neural Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network calculations involving tensor products are inefficient due to the sparsity of weights and activations, leading to unnecessary computations when many products equal zero.
Innovation Solution
Implement intra-tile and inter-tile preprocessing of weight tensors to create pre-processed tensors, utilizing lookahead and lookaside movements to identify and skip empty rows, and use tile sparsity maps to optimize computation cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If every element-wise product is calculated in tensor operations, then complete computation accuracy is maintained, but significant computational inefficiency occurs due to sparsity
Solution Approach 1:
The patent applies preliminary action by performing preprocessing of weight tensors before the main computation. Specifically, it identifies and marks empty rows in weight tiles ahead of time using sparsity maps, allowing the computation engine to skip these rows during actual tensor multiplication, thereby avoiding unnecessary computations and improving efficiency
Solution Approach 2:
The patent extracts and removes empty rows from weight tiles by creating sparsity maps that identify positions of empty rows. The computation engine then uses these sparsity maps to skip computations involving empty rows, effectively taking out unnecessary computational operations from the processing pipeline
2Productivity
If weight tensors are preprocessed to identify empty rows, then computation cycles are reduced, but device complexity increases due to preprocessing requirements
Solution Approach 1:
The patent segments the weight tensor into multiple weight tiles and processes each tile independently to identify empty rows. This segmentation allows the preprocessing to be done in a structured, modular manner, reducing the overall complexity by breaking down the large tensor into manageable chunks that can be processed separately
Solution Approach 2:
The patent introduces sparsity maps as intermediary data structures that store information about empty rows in weight tiles. These sparsity maps act as mediators between the weight tensors and the computation engine, allowing the engine to skip empty rows without requiring complex logic to detect them during computation
3Loss of time
If intra-tile and inter-tile preprocessing is performed, then the number of computation cycles is reduced, but the system complexity increases
Solution Approach 1:
The patent divides the preprocessing into two segments: intra-tile preprocessing that identifies empty rows within individual weight tiles, and inter-tile preprocessing that handles empty rows across multiple tiles. This segmentation allows each preprocessing stage to focus on a specific scope, reducing the complexity of each individual stage while achieving comprehensive optimization
Solution Approach 2:
The patent performs both intra-tile and inter-tile preprocessing before the main computation to identify and mark all empty rows in advance. This preliminary action ensures that the computation engine receives fully optimized weight tiles with complete sparsity information, maximizing the reduction in computation cycles
Data Source
AI summary
A system and method for weight preprocessing. In some embodiments, the method includes performing intra-tile preprocessing of a first weight tensor to form a first pre-processed weight tensor, and performing inter-tile preprocessing of the first pre-processed weight tensor, to form a second pre-processed weight tensor. The intra-tile preprocessing may include moving a first element of a first weight tile of the first weight tensor by one position, within the first weight tile, in a lookahead direction or in a lookaside direction. The inter-tile preprocessing may include moving a first row of a weight tile of the first pre-processed weight tensor by one position in a lookahead direction or by one position in a lookaside direction.