Replica Layers for Loss-Aware Neural Network Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in maintaining accuracy when weight quantization reduces the number of weight values, particularly in layers with fewer weights, leading to a loss of complexity and accuracy.
Innovation Solution
The use of replica layers with different sets of allowed weight values for each layer, allowing for more possible weight values while ensuring a threshold percentage of zero values, and employing techniques like ADMM and loss-aware penalty terms to constrain weights during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If weight quantization is applied to reduce the number of weight values, then device complexity is reduced and ease of manufacture is improved, but network accuracy is lost particularly in layers with fewer weights
Solution Approach 1:
The patent segments layers with fewer weights into multiple replica layers, where each replica layer uses standard quantization (3 allowed values) but the combined output of multiple replicas effectively provides higher precision. This segmentation allows the network to maintain accuracy in critical layers while still benefiting from quantization elsewhere.
Solution Approach 2:
The patent changes the parameter of allowed weight values dynamically: layers with sufficient weights use 3 allowed values {0, α, -α}, while layers with fewer weights are replicated to effectively provide more possible weight combinations. This parameter change resolves the contradiction by adapting quantization strategy to layer-specific requirements.
2Adaptability or versatility
If the number of allowed weight values is restricted to three per layer, then the network can be executed on specialized inference circuits, but the complexity of the network is reduced too far in some cases
Solution Approach 1:
The patent segments the network into standard layers and replicated layers. Standard layers maintain 3 allowed values for circuit compatibility, while replicated layers (composed of multiple standard layers) effectively increase complexity where needed. This segmentation allows compatibility with specialized circuits while preserving necessary network complexity.
Solution Approach 2:
The patent merges multiple replica layers (each with 3 allowed values) to create an effective layer with higher complexity. By combining the outputs of multiple simple layers, the network achieves higher effective complexity in specific layers without requiring any single layer to exceed the 3-value constraint, thus maintaining circuit compatibility.
3Manufacturing precision
If replica layers are used to increase the number of possible weight values, then network accuracy is maintained, but the device complexity increases
Solution Approach 1:
The patent applies replica layers selectively only to layers with fewer weights that require higher precision, while leaving other layers as standard single layers. This local application of replication minimizes the increase in device complexity while maintaining accuracy only where it is critically needed.
Data Source
AI summary
Some embodiments provide a method that receives a network with trained floating-point weight values. The network includes layers of nodes, each of which computes an output value based on input values and trained weight values. To replace a first layer of the trained network in a modified network with quantized weight values, the method defines multiple replica layers. Each replica layer includes nodes that correspond to nodes of the first layer, has a different set of allowed quantized weight values, and receives the same input values from a previous layer of the modified network such that groups of corresponding nodes from the replica layers operate correspondingly to the first layer. The method trains the quantized weight values of the modified network using a loss function with terms that account for effect on the loss function due to the quantization and for interactions between corresponding weight values of the replica layers.