Replica Layer Batch Normalization for Neural Network Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in maintaining accuracy when weight quantization reduces the number of weight values, particularly in layers with fewer weights, leading to a loss of complexity and accuracy.
Innovation Solution
The use of replica layers with different sets of allowed weight values for each layer, allowing for more possible weight values while ensuring a threshold percentage of zero values, and employing techniques like ADMM and loss-aware penalty terms to constrain weights during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weight quantization is applied to reduce the number of weight values, then the network can be executed efficiently on specialized inference circuits, but the network accuracy and complexity are reduced
Solution Approach 1:
The patent divides a single quantized layer into multiple replica layers, each with its own set of quantized weights. This segmentation allows the network to maintain higher accuracy by distributing the computational task across multiple layers while still using quantized weights for efficient execution on specialized circuits.
Solution Approach 2:
The patent combines multiple replica layers to effectively increase the number of available weight values. By merging the outputs of multiple replica layers, the system achieves finer-grained analysis capability while maintaining compatibility with quantized execution on specialized inference circuits.
2Adaptability or versatility
If the number of weight values is reduced through quantization, then the network can run on specialized inference circuits, but the percentage of zero weights decreases
Solution Approach 1:
By segmenting the quantized layer into multiple replica layers, each layer can independently maintain its sparsity pattern with a high percentage of zero weights. This segmentation preserves the quantity of zero weights while enabling execution on specialized circuits that require quantized inputs.
3Reliability
If replica layers are used to increase the number of possible weight values, then the network accuracy is maintained, but the device complexity increases
Solution Approach 1:
The patent creates replica layers that are copies of the original quantized layer, each with its own set of quantized weights. This copying approach maintains network accuracy by providing multiple pathways for computation while keeping each individual replica layer simple and manageable in complexity.
4Adaptability or versatility
If multiple copies of network layers are used to increase possible parameter values, then the allowed values set expands, but the training complexity increases
Solution Approach 1:
The patent changes the parameters of replica layers by applying different scaling factors to each layer's weights. This allows each replica layer to contribute differently to the final output, expanding the effective number of parameter values while maintaining a relatively simple training process through standardized quantization procedures.
Data Source
AI summary
Some embodiments provide a method for training a network including layers that each includes multiple nodes. The method identifies a set of related layers of the network. Each node in one of the related layers has corresponding nodes in each of the other related layers. Each set of corresponding nodes receives a same set of inputs and applies different sets of weights to the inputs to generate an output. The method identifies an element-wise addition layer including nodes that each add outputs of a different set of corresponding nodes from the related layers to generate a sum. The method uses a set of outputs generated by the nodes of each related layer to determine batch normalization parameters specific to each layer of the set of related layers. The method uses data generated by the element-wise addition layer to determine batch normalization parameters for the set of related layers.