Neural Network Compression via Block Segmentation and Adaptive Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network compression methods, such as structured pruning, require extensive sensitivity analysis for each operation layer, leading to high computational costs and neglect coupling effects between layers, with compression ratios often determined by experience rather than model loss.
Innovation Solution
The method divides neural network parameters into blocks, applies different compression ratios to each block, and updates weighting factors based on model loss through forward and backward propagation, selecting the operation branch with the maximum updated value as the compressed block, thereby balancing accuracy and model size reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional structured pruning methods are used to compress neural networks, then compression ratio can be achieved, but computational costs increase due to extensive sensitivity analysis required for each operation layer
Solution Approach 1:
The patent segments the neural network into multiple operation blocks (first operation block, second operation block, etc.), where each block contains multiple operation layers. This segmentation allows the method to compress different blocks with different compression ratios simultaneously, reducing the need for extensive sensitivity analysis across all layers while maintaining effective compression.
Solution Approach 2:
The patent applies local quality by determining different compression ratios for different operation blocks based on their specific characteristics and coupling effects. Instead of uniform compression, the method calculates compression ratios adaptively for each block, considering the coupling relationships between blocks, thereby reducing computational costs while maintaining compression effectiveness.
2Quantity of substance
If traditional compression methods compress each operation layer independently, then compression can be applied, but coupling effects between layers are neglected leading to suboptimal compression
Solution Approach 1:
The patent merges the compression process into operation blocks that contain multiple operation layers, treating them as coupled units rather than independent layers. The method calculates compression ratios for each block considering the coupling effects between layers within the block, and uses weight factors to combine the compressed blocks, thereby maintaining accuracy while achieving effective compression.
Solution Approach 2:
The patent implements feedback mechanisms where the compression ratio for each operation block is adjusted based on the coupling effects and performance evaluation. The weight factors are updated through forward and backward propagation, allowing the system to learn from the impact of compression on network performance and adjust compression strategies accordingly to minimize accuracy loss.
3Ease of manufacture
If compression ratios are determined by experience rather than model loss, then compression can be performed, but optimal compression ratios cannot be automatically determined
Solution Approach 1:
The patent enables self-service by allowing the neural network compression system to automatically determine optimal compression ratios for each operation block based on model loss and coupling effects. The method uses weight factors that are automatically updated through forward and backward propagation, eliminating the need for manual experience-based determination and achieving automated optimization.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting compression ratios as parameters based on model loss and coupling effects. The weight factors serve as adjustable parameters that are updated through propagation processes, allowing the system to automatically optimize compression ratios without manual intervention, thereby achieving both simplicity and automation.
Data Source
AI summary
A method for compressing a neural network includes: obtaining a neural network including a plurality of parameters to be compressed; dividing the parameters into J blocks; compressing a jth block with Kj compression ratios to generate Kj operation branches; obtaining Kj weighting factors; replacing the jth block with the Kj operation branches weighted by the Kj weighting factors to generate a replacement neural network; performing forward propagation to the replacement neural network, a weighted sum operation being performed on Kj operation results generated by the Kj operation branches with the Kj weighting factors and a result of the operation being used as an output; performing backward propagation to the replacement neural network, updated values of the Kj weighting factors being calculated based on a model loss; and determining an operation branch corresponding to the maximum value of the updated values of the Kj weighting factors as a compressed jth block.