Normalizing Flow Training with Self-Normalized Gradient Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing normalizing flows are restricted by triangular weight matrices, limiting their learning capabilities and computational efficiency, especially when dealing with high-dimensional data like images or audio signals.
Innovation Solution
Training an unrestricted normalizing flow with a method that approximates matrix inversions using iterative optimization algorithms, allowing for fully connected or convolutional layers without requiring matrix inversion, thereby reducing computational complexity from cubic to quadratic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If triangular weight matrices are used in normalizing flows, then computational complexity of matrix inversion is reduced to quadratic, but the learning capability and degrees of freedom are severely restricted
Solution Approach 1:
The patent segments the matrix inversion problem into two parts: (1) pre-computing and storing the inverse of the triangular matrix component, and (2) efficiently computing the inverse of the remaining unrestricted matrix component using the stored inverse. This segmentation allows the system to benefit from both triangular matrix efficiency and unrestricted matrix flexibility.
Solution Approach 2:
The patent performs preliminary computation and storage of the inverse of the triangular weight matrix component during the training phase. This pre-computed inverse is then reused during forward and backward passes, eliminating the need to re-compute it and thereby reducing overall computational complexity while allowing unrestricted weight matrices.
2Adaptability or versatility
If unrestricted weight matrices are used in normalizing flows, then learning capability and degrees of freedom are improved, but computational complexity of matrix inversion increases to cubic
Solution Approach 1:
The patent segments the unrestricted weight matrix into a triangular component and a remaining component. By segmenting the matrix, the system can apply efficient triangular matrix inversion techniques to part of the matrix while maintaining the flexibility of unrestricted weights in the remaining portion.
Solution Approach 2:
The patent changes the parameter representation by storing the inverse of the triangular component as a separate parameter. This parameter transformation allows the system to work with unrestricted weight matrices during training while efficiently computing gradients using the pre-stored inverse, thereby reducing computational complexity from cubic to quadratic.
3Measurement precision
If matrix inversion is computed during training, then accurate gradients can be obtained, but training time and computational resources are excessive
Solution Approach 1:
The patent performs the computationally expensive matrix inversion operation in advance during a preprocessing step, storing the result for later use. This preliminary action ensures that accurate gradients can be computed during training without repeatedly performing full matrix inversions, thereby significantly reducing training time while maintaining gradient accuracy.
Solution Approach 2:
The patent creates a copy of the inverse matrix and stores it for reuse during training. Instead of computing the inverse repeatedly during each training iteration, the system uses the pre-computed copy, which maintains numerical accuracy while dramatically reducing the computational burden and training time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Computer-implemented method for training a normalizing flow (60), wherein the normalizing flow (60) is configured to determine a first output signal (y) characterizing a likelihood or a log-likelihood of an input signal (x), wherein the normalizing flow (60) comprises at least one first layer, wherein the first layer comprises trainable parameters and a layer input to the first layer is based on the input signal (x) and the first output signal (y) is based on a layer output of the first layer, wherein training the normalizing flow comprises the steps of: • Determining at least one training input signal (xi); • Determining a training output signal (yi) for each training input signal (xi) by means of the normalizing flow (60); • Determining a first loss value, wherein the first loss value is based on a likelihood or a log-likelihood of the at least one determined training output signal (yi) with respect to a predefined probability distribution; • Determining an approximation of a gradient of the trainable parameters of the first layer with respect to the first loss value, wherein the gradient is dependent on an inverse of a matrix of the trainable parameters and determining the approximation of the gradient is achieved by optimizing an approximation of the inverse; • Updating the trainable parameters of the first layer based on the approximation of the gradient.