Neural Network Parameter Updating With Periodic Inverse Matrix Refresh
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational complexity of inverting the additional matrix in neural network models during second-order optimization significantly affects the training speed of neural networks.
Innovation Solution
A step-based and block-based update method is introduced, where the inverse matrix of the additional matrix is updated once per multiple iterations, rather than in each iteration, and neurons are divided into blocks for selective updating based on trace and sampling probabilities, reducing the frequency and scope of matrix inversions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the inverse matrix of the additional matrix is updated in each iteration during second-order optimization, then the training accuracy of the neural network model is improved, but the computational complexity and training time increase significantly
Solution Approach 1:
The patent applies periodic action by updating the inverse matrix of the additional matrix only once per iteration period (which includes multiple iterations), rather than updating it in every iteration. This periodic update strategy reduces the frequency of computationally expensive matrix inversion operations while still maintaining effective training, thereby resolving the contradiction between training accuracy and training time.
Solution Approach 2:
The patent segments the training process into multiple iteration periods, where each period contains multiple iterations. Within each period, the inverse matrix is updated only once, creating a structured pattern of update operations that balances computational load and training effectiveness.
2Reliability
If the inverse matrix of the additional matrix is updated in each iteration, then the model convergence is improved, but the computational complexity of matrix inversion significantly affects training speed
Solution Approach 1:
The patent implements periodic action by establishing iteration periods where the inverse matrix update occurs only once per period rather than every iteration. This approach maintains model convergence reliability while significantly improving training speed by reducing the frequency of computationally intensive matrix inversion operations.
Solution Approach 2:
The patent applies partial action by performing the inverse matrix update less frequently than the conventional approach (once per period instead of every iteration). This partial updating strategy is sufficient to maintain model convergence while avoiding the excessive computational complexity that would otherwise slow down training.
3Measurement precision
If second-order optimization algorithm is used to update parameters, then the optimization performance is improved, but the computational complexity of computing the inverse matrix increases
Solution Approach 1:
The patent resolves the contradiction between optimization performance and computational complexity by applying periodic action - the inverse matrix is computed and updated only once per iteration period rather than every iteration. This maintains the benefits of second-order optimization while dramatically reducing the overall computational complexity of the training process.
Solution Approach 2:
The patent segments the optimization process into iteration periods, allowing the computationally complex inverse matrix computation to be performed once per period rather than continuously. This segmentation reduces the cumulative computational complexity while preserving the optimization performance benefits of second-order methods.
Data Source
AI summary
The present disclosure relates to parameter updating methods. In one example method, a parameter in a neural network model is updated for a plurality of times through a plurality of iterations. The plurality of iterations include a first iteration period and a second iteration period. In the first iteration period, an inverse matrix of an additional matrix of the neural network model is updated once based on a quantity of iterations indicated by a first update stride. In the second iteration period, the inverse matrix of the additional matrix of the neural network model is updated once based on a quantity of iterations indicated by a second update stride, where the first iteration of the second iteration period is after the last iteration of the first iteration period in an iteration sequence, and the second update stride is greater than the first update stride.