Variational Autoencoder Training With Sigma-Point Latent Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training variational autoencoders (VAEs) requires high-variance policy gradients, necessitating numerous samples to accurately estimate mean and covariance, leading to inefficient training times, especially in high-dimensional spaces.
Innovation Solution
The method employs sigma points to determine a plurality of fixed positions relative to the latent distribution, using them for training without reparameterization, and adapts the machine learning system based on a loss function that accounts for differences between input and output signals, incorporating terms for variance and covariance projection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If reparameterization trick is used to simplify approximate posterior sampling, then gradient computation becomes straightforward, but gradient variance becomes high requiring numerous samples
Solution Approach 1:
The patent introduces sigma points as intermediary elements that mediate between the encoder output and the loss computation. These sigma points serve as deterministic samples that capture the distribution characteristics without requiring high-variance gradient estimates, thus resolving the contradiction between computational simplicity and gradient reliability
Solution Approach 2:
The patent transforms the sampling approach by changing from random sampling to deterministic sigma point sampling. This parameter change in the sampling strategy maintains gradient computation simplicity while significantly reducing gradient variance through the structured placement of sigma points around the mean
2Ease of operation
If random samples are taken in high dimensional latent space, then sampling is simple, but samples can be very far from the mean producing poor estimates
Solution Approach 1:
The patent performs preliminary action by pre-computing the sigma points based on the mean and covariance from the encoder. This preliminary structuring of sample points ensures they are positioned optimally to capture the distribution characteristics before the actual sampling and loss computation, improving estimation accuracy while maintaining simplicity
Solution Approach 2:
The patent addresses high-dimensional sampling challenges by transforming the sampling strategy from uniform random sampling to sigma point sampling that considers the distribution structure across multiple dimensions. This dimensional awareness through sigma points ensures better coverage and accuracy in high-dimensional latent spaces
3Measurement precision
If 2n + 1 training samples are used to accurately estimate mean and covariance with sigma points, then estimation accuracy improves, but training time increases
Solution Approach 1:
The patent enables self-service by having the encoder automatically provide the mean and covariance information needed to generate the sigma points. This self-contained approach eliminates the need for extensive external sampling, achieving accurate estimation with fewer samples and reduced training time
Solution Approach 2:
By pre-computing sigma points from the encoder's mean and covariance estimates, the system performs the necessary preparation work before the main training loop. This preliminary action reduces the computational burden during training while maintaining accurate estimation of distribution parameters
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Computer-implemented method for training a machine learning system (60), wherein the machine learning system (60) is configured to accept a sensor signal as input for anomaly detection and/or wherein the machine learning system (60) is configured for sampling a trajectory of a traffic participant and/or wherein the machine learning system (60) is configured for sampling of sensor signals and/or wherein the machine learning system (60) is configured for determining a value characterizing a likelihood of a sensor signal with respect to a training dataset, wherein training comprises: • Determining, by an encoder (61) of the machine learning system (60) and based on a training sensor signal (xi), a first intermediate representation (r1) characterizing a mean of a latent distribution of a latent space (l) and a second intermediate representation (r2) characterizing a variance and/or covariance of the latent distribution; • Determining, based on the first intermediate representation (r1) and the second intermediate representation (r2), a plurality of sigma points (σ) with respect to the latent distribution; • Determining an output signal (yi), wherein the output signal (yi) is determined by providing a randomly sampled sigma point (σ) of the plurality of sigma points (σ) to a decoder (62) of the machine learning system (60); • Adapting the machine learning system (60) based on a loss value, wherein the loss value is obtained from a loss function that comprises a term that characterizes a difference between the training sensor signal (xi) and the output signal (yi) the method being characterized in the loss function comprising a further term characterizing a difference of the output signal (yi) and a value sampled at random from a Normal distribution or wherein the term characterizes a likelihood of the output signal (yi) with respect to the Normal distribution, wherein further a mean of the Normal distribution is obtained by forward propagating the first intermediate representation (r1) through the decoder (62) and wherein a covariance matrix is determined by multiplying a Jacobian of the mean of the Normal distribution with the second representation and multiplying the result with a transpose of the Jacobian.