Deep Neural Network Compression With Sparse Weight Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require large memory storage, significant power consumption, and long inference times, making them impractical for deployment in many applications due to their large number of parameters and tendency towards over-fitting.
Innovation Solution
A method for compressing DNNs using an optimization model that prunes redundant connections while maintaining accuracy, allowing for sparse weights and controlling compression rates, applicable to various neural network architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks use many parameters to achieve high accuracy, then classification accuracy is improved, but memory storage requirements increase
Solution Approach 1:
The patent extracts and removes redundant weights from the neural network through iterative pruning. By identifying and eliminating unnecessary connections between neurons while preserving essential functionality, the network achieves significant compression (up to 1000x reduction) while maintaining classification accuracy above 95% on benchmark datasets.
Solution Approach 2:
The patent transforms the network weight parameters from dense floating-point values to sparse representations using structured pruning patterns. This parameter transformation enables the network to maintain accuracy with fewer effective parameters, directly reducing memory storage requirements while preserving computational functionality.
2Measurement precision
If deep neural networks use many parameters to achieve high accuracy, then classification accuracy is improved, but power consumption increases
Solution Approach 1:
The patent removes redundant computational operations and weights from the network, directly reducing the number of multiply-accumulate operations required during inference. This extraction of unnecessary computations leads to lower power consumption while maintaining accuracy through preservation of essential neural pathways.
Solution Approach 2:
By changing the parameter representation from dense to sparse formats with structured pruning patterns, the patent reduces computational complexity and memory access requirements, which directly translates to lower power consumption on mobile and edge devices while maintaining classification performance.
3Measurement precision
If deep neural networks use many parameters to achieve high accuracy, then classification accuracy is improved, but inference time increases
Solution Approach 1:
The patent extracts and eliminates redundant computational operations through iterative weight pruning. By removing unnecessary neurons and connections while preserving the essential decision-making pathways, the network achieves faster inference times with reduced computational complexity while maintaining high classification accuracy.
Solution Approach 2:
The patent transforms weight parameters into sparse representations with structured patterns that enable more efficient computation. This parameter change allows for optimized memory access patterns and reduced computational operations, directly decreasing inference time while preserving accuracy through careful preservation of critical weights.
4Quantity of substance
If deep neural networks are compressed to reduce memory storage, then memory storage is reduced, but classification accuracy decreases
Solution Approach 1:
The patent implements self-service through automated iterative pruning where the network identifies and removes its own redundant weights based on importance metrics. This self-optimizing process automatically determines the optimal compression level while preserving accuracy-critical connections, eliminating the need for manual intervention and achieving both compression and accuracy preservation simultaneously.
Solution Approach 2:
The patent uses feedback loops where pruning operations are followed by accuracy evaluation, and the results inform subsequent pruning decisions. This iterative feedback process allows the system to adaptively remove redundant weights while monitoring and maintaining classification accuracy, achieving optimal compression without sacrificing performance.
Data Source
AI summary
In an approach for compressing a neural network, a processor receives a neural network, wherein the neural network has been trained on a set of training data. A processor receives a compression ratio. A processor compresses the neural network based on the compression ratio using an optimization model to solve for sparse weights. A processor re-trains the compressed neural network with the sparse weights. A processor outputs the re-trained neural network.