Unstructured Neural Network Pruning via Tag Identifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks consume excessive memory and computational resources, making them inefficient for deployment on mobile devices due to their dense weight matrices and high computational complexity, which slows down other applications and increases costs.
Innovation Solution
Unstructured pruning of neural networks using binary masks and tag identifiers to reduce the number of active weight values, thereby creating a sparse weight matrix that reduces computational complexity and resource usage without significantly affecting accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dense weight matrices are used in deep neural networks, then computational accuracy is maintained, but memory consumption and computational complexity increase significantly
Solution Approach 1:
The patent extracts and removes redundant weight values from the dense weight matrices through unstructured pruning, retaining only the essential non-zero elements that contribute to computational accuracy. This extraction process transforms dense matrices into sparse matrices, significantly reducing memory consumption while preserving the functional integrity of the neural network.
Solution Approach 2:
The patent changes the density parameter of weight matrices from dense (all non-zero elements) to sparse (selected non-zero elements only). By modifying this fundamental parameter through pruning operations, the system reduces memory requirements and computational complexity while maintaining accuracy through careful selection of preserved weight values.
2Reliability
If dense weight matrices are used in deep neural networks, then computational accuracy is maintained, but computational resources and processing time increase
Solution Approach 1:
The patent extracts and removes redundant weight values from the dense weight matrices through unstructured pruning, retaining only the essential non-zero elements that contribute to computational accuracy. This extraction process transforms dense matrices into sparse matrices, significantly reducing memory consumption while preserving the functional integrity of the neural network.
Solution Approach 2:
The patent changes the density parameter of weight matrices from dense (all non-zero elements) to sparse (selected non-zero elements only). By modifying this fundamental parameter through pruning operations, the system reduces memory requirements and computational complexity while maintaining accuracy through careful selection of preserved weight values.
3Quantity of substance
If structured pruning is applied to neural networks, then resource requirements are reduced, but functional performance and accuracy are significantly affected
Solution Approach 1:
The patent extracts and removes redundant weight values from the dense weight matrices through unstructured pruning, retaining only the essential non-zero elements that contribute to computational accuracy. This extraction process transforms dense matrices into sparse matrices, significantly reducing memory consumption while preserving the functional integrity of the neural network.
Solution Approach 2:
The patent changes the density parameter of weight matrices from dense (all non-zero elements) to sparse (selected non-zero elements only). By modifying this fundamental parameter through pruning operations, the system reduces memory requirements and computational complexity while maintaining accuracy through careful selection of preserved weight values.
Data Source
AI summary
Embodiments provide methods and systems for unstructured pruning of a neural network. Method performed by a neural network pruning system includes accessing a trained neural network to be pruned. The trained neural network includes one or more neural layers. The method includes computing values of layer parameters for a filter associated with a neural layer based, at least in part, on a pruning criteria. The method further includes computing a tag identifier associated with the filter of the trained neural network based, at least in part, on corresponding values of layer parameters of the filter. The method further includes storing the tag identifier and the values of the layer parameters for filter of the trained neural network in a database.