Layerwise Fisher Approximation for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational complexity of calculating the Fisher Information Metric (FIM) for deep neural networks hinders the applicability of natural gradient descent (NGD), requiring additional backward passes and matrix inversions that are computationally expensive.
Innovation Solution
The system uses layerwise Fisher approximations to improve the quality of parameter updates in neural networks, eliminating the need for an additional backward pass and reducing computational overhead by performing local updates for each layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If natural gradient descent is used to improve training performance, then the quality of parameter updates is improved, but the computational complexity increases due to additional backward passes and matrix inversions
Solution Approach 1:
The patent divides the Fisher Information Matrix calculation into layer-wise independent computations. Each layer's contribution to the FIM is calculated separately using local gradients, avoiding the need for a single complex global backward pass. This segmentation reduces computational complexity while maintaining the quality improvements of NGD.
Solution Approach 2:
The patent uses local gradients computed at each layer to approximate the Fisher Information Matrix, rather than requiring global backward passes through the entire network. This local approach maintains the precision benefits of NGD while significantly reducing computational overhead by performing operations locally at each layer independently.
2Productivity
If the true Fisher Information Metric is calculated to achieve accurate natural gradient descent, then the training efficiency is improved, but the computational cost doubles due to requiring two backward passes
Solution Approach 1:
The patent performs preliminary local gradient computations at each layer during the forward pass, storing these local gradients for later use in the FIM approximation. This preliminary action eliminates the need for additional backward passes, as all necessary gradient information is already computed and stored from the initial forward propagation.
Solution Approach 2:
The patent makes each layer self-sufficient by computing its own local gradients and contributing to the FIM approximation independently. This self-service approach allows parallel computation across layers without requiring sequential backward passes, reducing computational cost while maintaining training efficiency.
3Reliability
If matrix inversion is performed for NGD updates to improve convergence, then the quality of training is improved, but the memory consumption increases for layers with large number of parameters
Solution Approach 1:
The patent segments the large-scale matrix inversion problem into smaller layer-wise independent matrix operations. By computing the FIM approximation and performing updates layer-by-layer using local gradients, the memory requirements are reduced from storing and inverting a single large global matrix to handling multiple smaller layer-specific matrices.
Solution Approach 2:
The patent changes the computational approach from exact matrix inversion to an approximate solution using local gradient information. This parameter change allows achieving convergence quality improvements without the full computational burden of exact NGD, reducing memory consumption by avoiding storage of complete Fisher Information Matrix for large layers.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using layer-wise Fisher approximations.