Neural Network Parameter Grouping for Adjustable Model Size Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network learning methods face challenges in reducing model size and calculation costs due to unnecessary parameters, and current methods like Group Lasso struggle to adjust the number of model parameters deleted, limiting the reduction in model size and calculation efficiency.
Innovation Solution
A neural network learning apparatus that includes a group parameter generating part, a regularization term calculating part, and a model updating part, which groups model parameters, calculates a regularization term based on defined distribution features, and updates the model to reduce the size while maintaining recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If Group Lasso regularization is applied to reduce model size, then the number of model parameters is reduced, but the ability to adjust the number of deleted parameters is limited
Solution Approach 1:
The patent introduces a mixture of Gamma distributions as the prior distribution for group parameters, where the shape parameter k and rate parameter θ are hyperparameters that can be adjusted. By changing these distribution parameters, the system can control the degree of sparsity and thus adjust the number of deleted parameters, resolving the limitation of fixed parameter reduction in Group Lasso.
Solution Approach 2:
The patent makes the regularization strength dynamic by introducing hyperparameters k and θ that can be adjusted based on data characteristics. The variational inference process dynamically adjusts the posterior distribution of group parameters, enabling adaptive control over which parameters are deleted and how many, rather than using a fixed threshold approach.
2Reliability
If more model parameters are retained to maintain recognition accuracy, then recognition accuracy is maintained, but calculation costs increase
Solution Approach 1:
The patent employs variational inference to compute the posterior distribution of group parameters based on the observed data and the prior distribution. This feedback mechanism allows the system to automatically determine the optimal balance between model complexity and data fit, retaining only the necessary parameters for accurate recognition while eliminating redundant ones that increase calculation costs.
Solution Approach 2:
By adjusting the hyperparameters k and θ of the Gamma distribution prior, the system can control the trade-off between model size and recognition accuracy. The variational inference process then optimizes the posterior parameters to achieve the best possible accuracy with the reduced number of parameters, efficiently managing calculation resources.
Data Source
AI summary
There is provided a neural network learning apparatus capable of adjusting an amount of reduction in a model size. A group parameter generating part grouping model parameters of a neural network model into arbitrarily defined groups and generating group parameters indicating features of the groups, a regularization term calculating part calculating a regularization term on an assumption that distribution of the group parameters is according to distribution defined by hyper parameters which are parameters defining distribution features, and a model updating part calculating a loss function from correct labels in teacher data, output probability distribution obtained by inputting feature values corresponding to the correct labels in the teacher data to the neural network model, and a regularization term, and updating the neural network model in a manner that a value of the loss function is decreased are included.