Weight Matrix Partitioning for Balanced Neural Network Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units face challenges in efficiently pruning weights while maintaining accuracy and enhancing training speed and efficiency.
Innovation Solution
A neural processing device with a weight manipulator that partitions two-dimensional weight matrices into preset sizes, generates pruning matrices, and transmits them to processing units, using SIMD width-based partitioning and threshold-based pruning to optimize weight removal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weights are pruned from the two-dimensional weight matrix, then training speed and hardware efficiency are enhanced, but accuracy may deteriorate
Solution Approach 1:
The two-dimensional weight matrix is divided into multiple partitioned matrices of equal size, where each partitioned matrix corresponds to a processing element in the PE array. This segmentation enables parallel processing while maintaining the ability to prune weights effectively within each partition without compromising overall model accuracy.
Solution Approach 2:
Different pruning operations are applied to different partitioned matrices based on their specific characteristics. Each partitioned matrix can have its own pruning threshold and sparsity level, allowing local optimization that maintains accuracy while improving training speed in each region of the weight matrix.
2Ease of manufacture
If the weight matrix is processed as a whole, then pruning can be applied uniformly, but processing time and computational complexity increase
Solution Approach 1:
The weight matrix is segmented into multiple partitioned matrices that can be processed independently and in parallel. Each partitioned matrix is handled by a separate processing element, enabling simultaneous pruning operations across all partitions, thereby reducing total processing time while maintaining uniform pruning criteria through centralized threshold generation.
Solution Approach 2:
The weight manipulator generates pruning thresholds and prepares pruning matrices before the actual training process begins. This preliminary preparation includes analyzing the weight distribution, determining appropriate thresholds, and creating the pruning structure in advance, so that during training, only the actual pruning and inference operations need to be performed, significantly reducing processing time.
3Productivity
If pruning threshold is increased to remove more weights, then sparsity and efficiency improve, but more zero groups become unbalanced across partitioned matrices
Solution Approach 1:
The system monitors the distribution of zero groups across partitioned matrices and uses this feedback to dynamically adjust pruning thresholds. When certain partitions have excessive zero groups while others have insufficient pruning, the threshold is adjusted to balance the distribution, ensuring uniform workload and efficient parallel processing while maintaining high overall sparsity.
Data Source
AI summary
A neural processing device and method for pruning thereof are provided. The neural processing device includes a processing unit configured to perform calculations, an L0 memory configured to store input and output data of the processing unit, wherein the input and output data include a two-dimensional weight matrix and a weight manipulator configured to receive the two-dimensional weight matrix and partition it into preset sizes to thereby generate partitioned matrices, to generate a pruning matrix by pruning the partitioned matrix, and to transmit the pruning matrix to the processing unit.