Transformer Neural Network Accelerator Using Mixed-Length Vector Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network pruning techniques require large indexing memory and increase operation complexity, posing challenges for hardware implementation due to high area and power consumption.
Innovation Solution
A sparsity-aware transformer neural network accelerator employing mixed-length vector pruning and a sparsity-aware accelerator architecture, utilizing a memory to store unpruned weights and inputs, and reconfigurable processing elements for efficient MAC operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If fine-grained pruning techniques are applied to reduce weight matrix size, then model lightweighting is achieved, but indexing memory requirements and operation complexity increase
Solution Approach 1:
The patent segments the weight matrix into multiple blocks and applies structured pruning at the block level rather than fine-grained element level. This segmentation allows pruning to be applied systematically across blocks, reducing the need for complex indexing mechanisms while achieving significant weight matrix size reduction. The block-based approach enables efficient memory management and reduces indexing overhead.
Solution Approach 2:
The patent changes the pruning parameter from fine-grained element-level sparsity to structured block-level sparsity. By modifying the pruning granularity parameter, the system achieves weight matrix size reduction while maintaining hardware-friendly operation patterns. This parameter change transforms the pruning approach to be compatible with accelerator architecture, reducing operation complexity.
2Quantity of substance
If conventional pruning techniques are implemented in hardware, then model compression is achieved, but hardware area and power consumption increase
Solution Approach 1:
The patent divides the neural network computation into multiple stages with pruning applied at intermediate block levels. This segmentation allows the hardware to process only non-zero elements at each stage, reducing the computational load and power consumption. The block-based segmentation enables efficient resource utilization in the hardware accelerator.
Solution Approach 2:
The patent applies pruning operations before the main computation phase, pre-processing the weight matrices to eliminate zero elements. This preliminary action reduces the data volume that needs to be processed in subsequent computation stages, thereby reducing hardware power consumption and energy usage during inference.
3Quantity of substance
If higher pruning ratios are applied, then model lightweighting is improved, but model performance degrades
Solution Approach 1:
The patent applies different pruning ratios to different blocks of the weight matrix based on their importance. Critical blocks retain higher density while less important blocks undergo more aggressive pruning. This local quality approach allows higher overall pruning ratios while maintaining model performance by preserving essential weight information in critical regions.
Solution Approach 2:
The patent employs dynamic pruning strategies where the pruning ratio is adjusted based on the specific characteristics of each weight block and layer. This dynamic approach allows the system to achieve higher pruning ratios in non-critical areas while maintaining performance by applying conservative pruning in critical areas, balancing compression and performance.
Data Source
AI summary
A neural network operation acceleration apparatus may comprise: a memory storing a mask matrix obtained by a first pruning process for a weight matrix of each layer of a transformer neural network; a plurality of reconfigurable processing elements performing multiply-and-accumulate (MAC) operations on the weight matrix to which the mask matrix is applied and the input of each layer; and a local adder tree summing the operation outputs of adjacent processing elements selectively based on direction strength information obtained by analyzing the mask matrix.