Variational Autoencoder Noise-Dependent Loss Stabilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The convergence of training in generative deep learning models, specifically variational autoencoders, deteriorates due to the fluctuating balance between restoration error and regularization terms caused by noise sampling, leading to instability in the learning process.
Innovation Solution
A modified loss function (L2) is introduced, where a correction coefficient based on the noise and standard deviation is applied to the regularization term, ensuring that both the restoration error and regularization term depend on the noise, stabilizing the training process by adding a noise-dependent normalization term.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If noise sampling is used in variational autoencoder training, then the model can learn latent representations, but the balance between restoration error and regularization terms fluctuates causing training convergence deterioration
Solution Approach 1:
The patent changes the parameter formulation of the loss function by introducing a correction coefficient that depends on the noise sampling variance. Specifically, the regularization term is multiplied by a correction coefficient calculated as the ratio of noise variance to latent variable variance, which stabilizes the loss value across different noise samples and prevents training convergence deterioration while maintaining latent representation learning capability
2Productivity
If a standard loss function is used, then training can proceed, but the balance between restoration error and regularization terms becomes unstable due to noise sampling
Solution Approach 1:
The patent implements feedback by calculating the correction coefficient based on the variance of the sampled noise and using it to adjust the regularization term in real-time during training. This feedback mechanism ensures that when noise variance is high, the regularization term is appropriately scaled, maintaining stable balance between restoration error and regularization terms throughout the training process
Data Source
AI summary
A non-transitory computer-readable recording medium stores a machine learning program causing a computer to execute a process including: calculating an average and a variance of a latent variable by inputting input data to an encoder; sampling a noise based on a normal distribution of the variance; calculating the latent variable by adding the noise to the average; calculating output data by inputting the calculated latent variable to a decoder; and training the encoder and the decoder in accordance with a loss function, the loss function including a value and an error between the input data and the output data, the value being obtained by multiplying encoding information by a correction coefficient based on the noise, the encoding information being information of a probability distribution of the latent variable and a prior distribution of the latent variable.