Neural Network Pruning for Hardware Accelerator Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators for neural networks do not effectively benefit from conventional pruning methods, as they randomly and indiscriminately remove synaptic weights, which can disrupt the full-connectivity and computational power of Hopfield Neural Networks (HNNs), leading to performance and power efficiency issues.
Innovation Solution
The implementation of a regularized pruning technique that systematically eliminates synaptic weights using a regularization function, such as L1 and L2 regularization, to customize the pruning process for the hardware implementation, specifically turning off entire columns in a memristor crossbar array to maintain full-connectivity and reduce power and area consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional pruning methods are used to reduce neural network size, then the network size is reduced, but the full-connectivity and computational power of Hopfield Neural Networks are disrupted
Solution Approach 1:
The patent segments the neural network into modular components (input layer, output layer, and pruned hidden units) that can be independently managed. By dividing the network structure, the system can remove specific hidden units while preserving the connectivity patterns within the remaining units, thus maintaining full-connectivity in the pruned network.
Solution Approach 2:
The patent applies local quality by selectively pruning specific hidden units based on their contribution to network performance rather than uniformly removing weights. The regularization function identifies and removes only the least important connections while preserving critical connectivity patterns, ensuring that the remaining network maintains full-connectivity properties where needed.
2Productivity
If the size of the neural network grows, then the computational power increases, but the matrix operations bottleneck increases (approximately 90% bottleneck)
Solution Approach 1:
The patent extracts and removes redundant or less important hidden units from the neural network using regularization-based pruning. By taking out these unnecessary components, the network size is reduced while maintaining essential computational capabilities, thereby reducing the matrix operation bottleneck without significantly compromising computational power.
Solution Approach 2:
The patent applies partial action by selectively pruning only the least important connections rather than removing a large portion of the network. This approach achieves sufficient reduction in matrix operation complexity while preserving the majority of computational power, avoiding excessive pruning that would harm performance.
3Productivity
If hardware accelerator resources are increased, then the computational performance improves, but the IC area and cost increase
Solution Approach 1:
The patent implements dynamic resource allocation by using regularization-based pruning to adaptively determine the optimal network size for hardware implementation. The pruning process dynamically identifies and removes redundant units, allowing the hardware accelerator to achieve high computational performance with minimal IC area by allocating resources only to essential network components.
Solution Approach 2:
The patent changes the network parameters (number of hidden units, connection weights) through systematic pruning to optimize the trade-off between computational performance and hardware resources. By adjusting these parameters to remove unnecessary elements, the system achieves efficient hardware utilization with reduced IC area while maintaining acceptable performance levels.
Data Source
AI summary
Systems and methods are provided for implementing hardware optimization for a hardware accelerator. The hardware accelerator emulates a neural network. Training of the neural network integrates a regularized pruning technique to systematically reduce a number of weights. A crossbar array included in hardware accelerator can be programmed to calculate node values of the pruned neural network to selectively reduce the number of weight column lines in the crossbar array. During deployment, the hardware accelerator can be programmed to power off periphery circuit elements that correspond to a pruned weight column line to optimize the hardware accelerator for power. Alternatively, before deployment, the hardware accelerator can be optimized for area by including a finite number of weight column line. Then, regularized pruning of the neural network selectively reduces the number of weights for consistency with the finite number of weight columns lines in the hardware accelerator.