Neural Network Regularization for Memory-Efficient Mobile Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying trained neural networks, particularly CNNs, on mobile devices with limited memory capabilities is challenging due to their high memory requirements, and existing methods for reducing memory footprint, such as parameter pruning and topology simplification, often impair performance or require separate training.
Innovation Solution
A novel regularization function that promotes low weight values by considering the sensitivity of the network output to weight variations, allowing for sparse weight distributions through a gradient descent process, followed by pruning weights below a threshold, thereby reducing memory needs without compromising performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard neural network training is used, then high classification performance is achieved, but memory requirements become too high for mobile devices
Solution Approach 1:
The patent modifies the cost function by introducing a regularization term that penalizes large weight values. This parameter change in the training objective function guides the network to learn sparse weight distributions, reducing the number of significant parameters that need to be stored in memory while preserving classification accuracy.
Solution Approach 2:
The patent applies pruning techniques to remove and eliminate redundant or less important connections (weights) from the neural network after training. By extracting and removing these unnecessary parameters, the memory footprint is reduced without significantly impacting the network's classification performance.
2Quantity of substance
If parameter pruning is applied to reduce memory footprint, then memory requirements decrease, but classification performance deteriorates
Solution Approach 1:
The patent performs preliminary training with regularization that anticipates future pruning. By pre-shaping the weight distribution during training to be sparse and robust, the network is prepared to withstand the removal of parameters without significant performance loss, making the pruning process more effective.
Solution Approach 2:
The patent employs iterative pruning and retraining processes where the network structure is refined based on performance feedback. After each pruning step, the network is retrained to recover performance, creating a feedback loop that gradually optimizes the balance between memory efficiency and classification accuracy.
3Quantity of substance
If topology simplification is used to reduce memory usage, then memory requirements decrease, but separate training is required
Solution Approach 1:
The patent merges the weight reduction objective directly into the standard training process through regularization. Instead of separating training and pruning into distinct phases with different topologies, the method combines parameter compression guidance with the primary classification learning objective, allowing both to proceed simultaneously in a unified training framework.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is provided. The method comprises: - providing a neural network having a set of weights (W) and being configured to receive an input data structure (x) for generating a corresponding output array (y(x,W)) according to values of said set of weights - training (200) the neural network (100) to obtain a trained neural network (100''), said training comprising setting values of the set of weights by means of a gradient descent algorithm which exploits a cost function comprising a loss term and a regularization term; - deploying the trained neural network (100'') on a device (620) through a communication network (650); - using the deployed trained neural network (100'') on the device, wherein: the regularization term is based on a rate of change of elements of the output array caused by variations of the set of weights values.