Deep Neural Network Compression With Skip Connections and Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks require significant storage and computational resources, limiting their deployment in resource-constrained environments such as mobile and embedded systems, and existing compression methods often compromise network performance.
Innovation Solution
Introduce bridging connections within the neural network architecture, followed by parameter and activation quantization, to reduce resolution and storage requirements while maintaining performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the resolution of parameters is reduced to decrease storage requirements, then storage space is saved, but network performance deteriorates
Solution Approach 1:
The neural network is segmented into multiple layers with different resolution requirements. The patent applies different quantization levels to different layers, allowing critical layers to maintain higher precision while non-critical layers use lower precision, thus preserving overall performance while reducing total storage requirements.
Solution Approach 2:
Different parts of the neural network are assigned different quality levels (resolution). The patent identifies which layers benefit most from high precision and which can tolerate lower precision, applying local quality optimization to maintain performance where needed while reducing storage where possible.
2Adaptability or versatility
If the resolution of parameters is reduced to enable deployment in resource-constrained environments, then adaptability improves, but computational precision deteriorates
Solution Approach 1:
The patent implements dynamic precision adjustment where the resolution of parameters can be adapted based on the specific deployment environment and task requirements. This allows the system to optimize between precision and resource usage dynamically rather than using a fixed precision level throughout.
Solution Approach 2:
The patent changes the precision parameter of neural network parameters based on deployment constraints. By systematically varying the precision level across different layers and operations, the system adapts to resource-constrained environments while maintaining sufficient computational precision for accurate results.
3Quantity of substance
If quantization is applied to reduce storage requirements, then storage efficiency improves, but information loss increases
Solution Approach 1:
The patent applies preliminary quantization analysis and layer importance assessment before final quantization. By pre-identifying which layers are most sensitive to quantization errors, the system can preserve more information in critical layers while applying more aggressive compression to non-critical layers, minimizing overall information loss.
Solution Approach 2:
The patent employs feedback mechanisms during the quantization process, using performance metrics to adjust quantization strategies. By monitoring the impact of quantization on network performance and adjusting precision levels accordingly, the system minimizes information loss while achieving storage efficiency goals.
Data Source
Figure 0
Figure 1
Figure 2
AI summary
The invention relates to a method for compressing a deep neural network (12). The deep neural network comprises a plurality of layers, each of which is connected at its input to the immediately preceding layer in the sequence according to a predefinable order. The method comprises the following steps: adding a skip connection (24). This is followed by reducing the resolution of the layer parameters. The invention further relates to a computer program and a device for executing the method.