Sparse Neural Network Training for Mobile Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying trained Convolutional Neural Networks (CNNs) on mobile devices with limited storage memory is challenging due to high memory requirements, which exceed the available memory on these devices.
Innovation Solution
A modified regularization function is introduced that promotes low values of weights in the neural network by considering the rate of change of the network output caused by variations in the weights, allowing for a reduced memory footprint without compromising performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a trained CNN is deployed on mobile devices, then object classification functionality is provided, but the memory requirements exceed the available memory on these devices
Solution Approach 1:
The patent extracts and removes redundant weights from the neural network model. By identifying and eliminating weights that contribute minimally to the network output (those with small absolute values), the model size is reduced while maintaining essential functionality. This extraction process directly addresses the memory occupation problem by removing unnecessary parameters from the deployed model.
Solution Approach 2:
The patent changes the parameter representation by transforming weight values into a sparse format. Through regularization techniques and thresholding operations, continuous weight parameters are converted into a sparse structure where many parameters become zero or near-zero. This parameter transformation reduces the effective memory footprint while preserving the network's classification capability.
2Quantity of substance
If the number of parameters in the neural network is reduced, then memory occupation decreases, but the performance of the network may be compromised
Solution Approach 1:
The patent applies partial action by selectively removing only a portion of the weights rather than eliminating all parameters. By using regularization to identify and prune only the least important weights (those with small magnitudes), the network retains sufficient parameters to maintain high classification accuracy. This partial removal strategy achieves parameter reduction without excessive pruning that would degrade performance.
Solution Approach 2:
The patent incorporates feedback mechanisms through regularization techniques that continuously monitor the impact of weight removal on network performance. During training and deployment, the system evaluates whether parameter reductions are affecting classification accuracy and adjusts the pruning strategy accordingly, ensuring that performance thresholds are maintained while maximizing parameter reduction.
Data Source
AI summary
A method includes providing a neural network having a set of weights. The neural network receives an input data structure for generating a corresponding output array according to values of the set of weights. The neural network is trained to obtain a trained neural network. The training includes setting values of the set of weights with a gradient descent algorithm which exploits a cost function including a loss term and a regularization term. The trained neural network is deployed on a device through a communication network, and used by the device. The regularization term is based on a rate of change of elements of the output array caused by variations of the set of weights values.