Programmable Thresholds for Neural Network Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current sparsity injection techniques in deep neural networks (DNNs) often lead to accuracy reduction and increased complexity, especially when not properly retrained with optimal hyperparameters, and are difficult to implement, particularly in emerging neural networks like transformers with non-ReLU based activations.
Innovation Solution
The introduction of programmable thresholds for pruning activations and weights in DNNs, allowing for iterative determination of optimal thresholds to minimize accuracy loss and enhance sparsity, which can be executed during the compilation stage or using existing sparsity hardware without additional overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If sparsity injection techniques are applied to reduce computation cost, then computational efficiency is improved, but model accuracy deteriorates
Solution Approach 1:
The patent introduces programmable thresholds as a new parameter for controlling sparsity injection. By adjusting these thresholds, the system can dynamically control the degree of sparsity in different network layers, allowing optimization of the trade-off between computational efficiency and model accuracy without requiring complete retraining.
Solution Approach 2:
The sparsity injection mechanism is made dynamic through programmable thresholds that can be adjusted during model deployment. This allows the system to adapt the sparsity level based on computational resources and accuracy requirements, transforming a static pruning approach into a dynamic one that can respond to different operational conditions.
2Loss of energy
If sparsity injection techniques are applied to reduce computation cost, then computational efficiency is improved, but implementation complexity increases
Solution Approach 1:
The patent performs sparsity analysis and threshold determination during the model compilation stage rather than during runtime execution. This preliminary action allows the complex sparsity injection logic to be prepared in advance, reducing the operational complexity during deployment and enabling simpler execution at inference time.
Solution Approach 2:
The patent introduces a sparsity encoder as an intermediary component that bridges the original model and the pruned output. This encoder handles the complex sparsity injection operations, isolating the complexity from the main computational graph and simplifying the overall implementation by separating the sparsity management function.
3Productivity
If existing sparsity hardware is used without additional overhead, then hardware efficiency is improved, but functionality is limited
Solution Approach 1:
The patent designs the sparsity injection mechanism to work universally across different neural network architectures and activation functions. The programmable threshold approach can be applied to ReLU, Sigmoid, GELU, SoftMax, and other activation functions, making the same hardware mechanism adaptable to various models without requiring architecture-specific optimizations.
Solution Approach 2:
By using programmable thresholds, the system can adapt existing hardware to handle different sparsity requirements and network architectures by simply changing the threshold parameters. This parameter-based adaptability allows the same hardware to serve multiple functions across different models without physical modification.
Data Source
AI summary
Activations (e.g., output activations) or weights of intermediate layers of deep neural networks (DNNs) can be pruned to increase sparsity and reduce the amount of computation required for performing the computations in the layers or subsequent layers. A pruning threshold may be determined, e.g., through an iterative process, and activations or weights having absolute values lower than the pruning threshold may be changed to zero. A first pruning threshold may be used to prune an output tensor or kernel of a layer. The loss in the accuracy of the DNN due to the pruning may be determined. A second pruning threshold may be determined based on the first pruning threshold and the accuracy loss. The DNN may be modified by adding a pruning operation to the layer. The pruning operation can prune output tensors or kernels of the layer based on the second pruning threshold.