Neural Network Coefficient Sparsity for Memory Footprint Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks with highly parameterised sets of coefficients face challenges in implementation due to large memory footprints and computational demands, making them difficult to deploy on devices with limited processing or memory resources.
Innovation Solution
A method of compressing sets of coefficients by applying sparsity to groups of coefficients and using a compression scheme aligned with these groups, representing each group with an integer number of compressed values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If highly parameterised sets of coefficients are used in neural networks, then accuracy and modeling capability are improved, but memory footprint and computational demands increase
Solution Approach 1:
The patent extracts and removes redundant or less important coefficients from the neural network model through pruning techniques. By identifying and eliminating coefficients that contribute minimally to the overall accuracy, the model achieves compression while maintaining performance, directly addressing the contradiction between accuracy and memory footprint.
Solution Approach 2:
Instead of starting with a small model and adding coefficients, the patent inverts the approach by starting with a large, accurate model and systematically removing coefficients. This inversion allows the model to begin at peak accuracy and then be compressed, ensuring that the final model maintains the highest possible accuracy for its size.
2Measurement precision
If highly parameterised sets of coefficients are used in neural networks, then accuracy and modeling capability are improved, but computational demands increase
Solution Approach 1:
The patent removes computationally expensive coefficients that contribute minimally to accuracy. By pruning these coefficients, the number of computations required during inference is reduced, directly lowering computational demands while preserving the model's accuracy on essential features.
Solution Approach 2:
The patent changes the parameters of the neural network by modifying the sparsity level and pruning thresholds. By adjusting these parameters, the model finds an optimal balance between accuracy and computational efficiency, allowing deployment on devices with varying computational capabilities.
3Quantity of substance
If sparsity is applied to coefficients, then memory footprint and computational demands are reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent applies sparsity selectively to different parts of the model based on their importance. By identifying which coefficients are critical to accuracy and which are redundant, the method applies sparsity locally rather than uniformly, preserving accuracy in critical regions while compressing non-critical regions.
Solution Approach 2:
The patent uses feedback mechanisms during the pruning process, where the model's accuracy is continuously monitored and used to adjust the sparsity level. This feedback loop ensures that sparsity is increased only up to the point where accuracy remains acceptable, preventing excessive compression that would harm performance.
4Power
If sparsity is applied to coefficients, then computational demands are reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent applies computational optimization selectively to different parts of the model. By identifying which coefficients are most important for accuracy, the method preserves full precision for these coefficients while applying sparsity and compression to less critical ones, maintaining overall accuracy while reducing computational demands.
Solution Approach 2:
The patent adjusts parameters such as sparsity thresholds and compression levels to find the optimal balance between computational efficiency and accuracy. By dynamically tuning these parameters based on the specific model and application requirements, the method achieves significant computational savings without unacceptable accuracy loss.
Data Source
AI summary
A computer implemented method of training a neural network configured to combine a set of coefficients with respective input data values. So as to train a test implementation of the neural network, sparsity is applied to one or more of the coefficients according to a sparsity parameter, the sparsity parameter indicating a level of sparsity to be applied to the set of coefficients; the test implementation of the neural network is operated on training input data using the coefficients so as to form training output data; in dependence on the training output data, assessing the accuracy of the neural network; the sparsity parameter is updated in dependence on the accuracy of the neural network; and a runtime implementation of the neural network is configured in dependence on the updated sparsity parameter.