Structured Pruning for Neural Network Weight Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large weight matrices in neural network models require significant memory and computational resources, limiting their usability and performance, especially in systems with lower memory and computational capabilities.
Innovation Solution
The structured pruning system applies a block-wise row and column pruning (BRCP) algorithm to partition and prune the weight matrix, converting it into a sparser output pattern that can be further compressed for efficient hardware implementation, ensuring that at least one sub-row in each row remains unpruned to maintain accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large weight matrix is used in the neural network model, then the model can achieve high recognition accuracy, but it requires significant memory and computational resources
Solution Approach 1:
The weight matrix is divided into multiple blocks, and each block is further divided into sub-blocks. This segmentation allows selective pruning of sub-columns and sub-rows while maintaining the overall structure, reducing memory requirements while preserving recognition accuracy through careful selection of what to retain.
Solution Approach 2:
The patent extracts and removes redundant or less important sub-columns and sub-rows from the weight matrix blocks. By identifying and eliminating these less critical components through pruning operations, the model achieves memory compression while maintaining the essential information needed for accurate recognition.
2Measurement precision
If a large weight matrix is used in the neural network model, then the model can achieve high recognition accuracy, but it incurs huge computational cost for multiplication and addition operations
Solution Approach 1:
By segmenting the weight matrix into blocks and sub-blocks, the patent enables selective pruning of sub-columns and sub-rows. This reduces the number of multiplication and addition operations required during inference, as fewer weight elements need to be processed, thereby lowering computational cost while maintaining accuracy.
Solution Approach 2:
The patent applies partial pruning by retaining at least one sub-row in each block rather than completely removing entire rows. This partial action approach sufficiently reduces computational operations while preserving enough information to maintain recognition accuracy, avoiding the need for excessive pruning that would harm performance.
3Quantity of substance
If the weight matrix is pruned to reduce memory overhead, then memory efficiency improves, but the model accuracy may deteriorate
Solution Approach 1:
The patent segments the weight matrix into blocks and sub-blocks, allowing fine-grained pruning at the sub-column and sub-row level. This segmentation enables memory efficiency improvements by removing redundant elements while preserving critical information in other sub-blocks, thereby maintaining model accuracy.
Solution Approach 2:
The patent employs partial pruning by ensuring that at least one sub-row is retained in each block. This partial action prevents complete removal of any row, thereby preserving sufficient information to maintain model accuracy while still achieving memory overhead reduction through pruning of less critical sub-columns and sub-rows.
4Quantity of substance
If traditional pruning methods are applied to the weight matrix, then memory overhead is reduced, but the computational complexity of maintaining accuracy constraints increases
Solution Approach 1:
The patent divides the weight matrix into blocks and sub-blocks, creating a hierarchical structure that simplifies the pruning process. This segmentation allows the algorithm to operate on smaller sub-columns and sub-rows independently, reducing the computational complexity of maintaining accuracy constraints compared to evaluating the entire weight matrix at once.
Solution Approach 2:
The patent applies partial pruning by retaining at least one sub-row per block, which provides a simple constraint rule that reduces algorithmic complexity. This partial action approach avoids the need for complex optimization algorithms while still maintaining accuracy, as the constraint is easy to enforce during the pruning process.
Data Source
AI summary
An input weight pattern of a machine learning model may be received. The input weight pattern may be pruned to produce an output weight pattern based on a predetermined pruning algorithm. The pruning algorithm may include partitioning the input weight pattern into a plurality of sub-patterns, each row of the input weight pattern including sub-rows of a first number of sub-patterns, and each column of the input weight pattern including sub-columns of a second number of sub-patterns; and pruning sub-columns and sub-rows from the plurality of sub-patterns to achieve predetermined column and row sparsities respectively, with a constraint that at least one sub-row in each row of the input weight pattern is not pruned. The output weight pattern may further be compressed to produce a compact weight pattern. The compact weight pattern has lower memory and computational overheads as compared to the input weight pattern for the machine learning model.