Neural Network Layer Tuning Under Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks often suffer from performance discrepancies and reduced accuracy due to training data mismatch between the training environment and deployment environment, necessitating adjustments without complete retraining, especially in systems with limited memory and computing power.
Innovation Solution
A method involving Fisher information analysis to determine the importance of each layer in a neural network, using a non-dominated genetic sorting algorithm to select a layer combination for tuning while respecting memory constraints, followed by parameter adjustment of the selected layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If complete retraining of the neural network is performed to improve accuracy, then the neural network accuracy is improved, but the memory and computing power requirements increase significantly
Solution Approach 1:
The patent segments the neural network into individual layers and analyzes each layer's contribution to output performance separately. By calculating a performance index for each layer based on Fisher information, the method identifies and retires only the specific layers that need improvement, rather than retraining the entire network. This segmentation enables partial retraining that reduces memory and computing resource requirements while maintaining accuracy improvements.
2Measurement precision
If the neural network is tuned by the deployed computer system to improve accuracy, then the neural network accuracy is improved, but the power consumption and memory usage increase
Solution Approach 1:
The patent divides the tuning process into layer-specific operations, calculating performance indices and identifying only the necessary layers for retuning. This segmentation reduces the computational burden and power consumption compared to full network retraining, enabling deployed systems with limited resources to perform accurate tuning locally without excessive energy expenditure.
3Measurement precision
If Fisher information analysis is performed on all layers to identify important layers, then the layer selection accuracy is improved, but the computational complexity increases
Solution Approach 1:
The patent applies Fisher information analysis to each layer independently and calculates a separate performance index for each layer. This segmented approach allows efficient identification of important layers without requiring complex global analysis, reducing computational complexity while maintaining accurate layer selection for retuning.
Solution Approach 2:
The patent extracts the performance index information from each layer using Fisher information calculation, separating the analysis into manageable layer-specific computations. This extraction method identifies critical layers efficiently by focusing computational resources on individual layer contributions rather than analyzing the entire network simultaneously.
Data Source
Figure 1~2
Figure 3~5
AI summary
According to one aspect, a computer-implemented method is proposed for analyzing and adjusting an already trained artificial neural network, the artificial neural network comprising a succession of layers, each layer presenting a parameter tensor, the method comprising: - an extraction (22) of Fisher information for each parameter of the artificial neural network, - a calculation (23) of an index for each layer of the artificial neural network, this index being representative of the Fisher information calculated for the parameters of this layer, - a definition (24) of a combination of layers to be adjusted of the artificial neural network, the combination of layers being defined from the parameter tensor indices of the layers of the artificial neural network, - a comparison (25) between the memory occupation required for the adjustment of the parameters of said combination of layers and a maximum memory occupation threshold.