Replica Layer Sparsity for Neural Network Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in maintaining accuracy when weight quantization reduces the number of weight values, particularly in layers with fewer weights, leading to a loss of complexity and accuracy.
Innovation Solution
The use of replica layers with different sets of allowed weight values for each layer, allowing for more possible weight values while ensuring a threshold percentage are zero, and employing techniques like ADMM and loss-aware penalty terms to constrain weights during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If weight quantization is applied to reduce the number of weight values, then the network can be executed on specialized inference circuits and complexity is reduced, but accuracy is lost particularly in layers with fewer weights
Solution Approach 1:
The network layers are segmented into replica layers, where each replica layer maintains separate weight values. This segmentation allows the system to preserve more weight information (improving accuracy) while still maintaining sparsity constraints for execution on specialized circuits. Instead of applying quantization uniformly across all layers, the segmentation enables differential treatment where replica layers collectively represent the original layer's functionality with enhanced precision.
Solution Approach 2:
The invention changes the parameter representation by allowing replica layers to have different sets of allowed weight values. Rather than constraining all weights to a fixed small set (e.g., {0, 1, -1}), the system allows each replica layer to have its own weight value set, and the combination of replica layers effectively increases the number of possible weight values. This parameter change enables finer-grained weight representation while maintaining sparsity.
2Productivity
If the number of allowed weight values is restricted to ensure sparsity, then execution on specialized circuits is enabled, but the number of possible weight values decreases leading to loss of complexity
Solution Approach 1:
The invention adds another dimension to the weight representation by introducing multiple replica layers. Instead of increasing the number of weight values within a single layer (which would violate sparsity constraints), the system distributes the weight representation across multiple layers in the vertical dimension. Each replica layer maintains its own sparsity pattern, but collectively they provide a richer weight value space, effectively increasing complexity without sacrificing execution efficiency.
3Reliability
If replica layers are used to increase possible weight values, then accuracy is maintained, but the number of layers and parameters increases
Solution Approach 1:
The replica layers are merged in the sense that they process the same input data and their outputs are combined to produce the final layer output. This merging approach allows the system to achieve enhanced accuracy (by having more weight value options) without completely independent training of each replica layer. The shared input processing and output combination create a form of efficiency that mitigates the increase in layer count.
Data Source
AI summary
Some embodiments provide a method for training multiple parameters of a machine-trained (MT) network subject to a sparsity constraint that requires a threshold portion of the parameters to be equal to zero. A first set of the parameters subject to the sparsity constraint are grouped into groups of parameters. For each parameter of a second set of the parameters subject to the sparsity constraint, the method determines an accuracy penalty associated with the parameter being set to zero. For each group of parameters in the first set of parameters, the method determines a minimum accuracy penalty for each possible number of parameters in the group being set to zero. The method uses the determined accuracy penalties to set to the value zero at least the threshold portion of the plurality of parameters.