Neural Network Model Reduction via Bias Combination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing model size-reduction techniques for machine learning models often result in decreased calculation efficiency and accuracy due to the removal of parameters with small influence, leading to the loss of useful information and degradation in model accuracy.
Innovation Solution
A method that identifies and deletes neurons without input or output connections in a neural network, compensates for biases by combining them with biases of connected neurons, and removes these neurons to reduce model size while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If parameters with small influence are deleted to reduce model size, then model size is reduced, but accuracy and calculation efficiency deteriorate
Solution Approach 1:
The patent extracts and removes only the most redundant parameters (those with zero or near-zero influence) while preserving parameters with small but non-zero influence. This selective extraction approach reduces model size while maintaining accuracy, as opposed to removing all parameters with small influence which would degrade performance.
Solution Approach 2:
The patent applies parameter thresholding and influence analysis to identify which parameters can be safely removed. By changing the criteria for parameter removal from a simple size-based approach to an influence-based approach, the model maintains accuracy while reducing size. The bias combination step also represents a parameter change strategy to preserve information during reduction.
2Quantity of substance
If parameters with small influence are deleted to reduce model size, then model size is reduced, but calculation efficiency deteriorates
Solution Approach 1:
The patent extracts only the most redundant parameters for removal, leaving behind parameters that contribute to calculation efficiency. By being selective about what is removed, the model maintains efficient computation while reducing overall size.
Solution Approach 2:
The patent changes the approach to parameter removal by using influence analysis to identify truly redundant parameters. This ensures that parameters important for calculation efficiency are preserved, while only parameters that would harm efficiency if kept are removed.
3Quantity of substance
If arbitrary parameter deletion is performed to reduce model size, then model size is reduced, but useful information is lost
Solution Approach 1:
The patent uses influence analysis as a feedback mechanism to evaluate the impact of each parameter before removal. This feedback loop ensures that only parameters with negligible influence are removed, preventing loss of useful information while achieving model reduction.
Solution Approach 2:
The patent changes the deletion strategy from arbitrary removal to influence-based selection. By analyzing parameter influence and using bias combination techniques, the patent ensures that information is preserved during reduction, preventing useful information loss.
Data Source
AI summary
A non-transitory computer-readable storage medium storing a model reduction program that causes at least one computer to execute a process, the process includes identifying as deletion targets a first neuron that does not connect to an input layer in a neural network; identifying as deletion targets a second neuron that does not connect to an output layer in a neural network; combining a bias of the first neuron with a bias of a third neuron connected to the first neuron on an output side; and deleting the first neuron and the second neuron from the neural network.