Denoising Autoencoder Training for Missing Structured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle with noisy and incomplete datasets, particularly in structured data, leading to inaccurate predictions and poor performance due to issues like data leakage, bias, and inability to handle outliers, which are exacerbated by the deletion or improper handling of missing values.
Innovation Solution
An enhanced denoising autoencoder (DAE) that predicts missing values by leveraging missing block statistics to generate noisy datasets, avoiding data duplication and using a sampling-based method to train the DAE, reducing processing power, training time, and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional machine learning models are trained on complete datasets by deleting rows with missing values, then training speed and convergence are improved, but data loss increases and model accuracy deteriorates due to bias and inability to handle real-world noisy data
Solution Approach 1:
The patent transforms the harmful effect of missing values and noisy data into a beneficial training mechanism by deliberately introducing artificial missing patterns during training. The denoising autoencoder learns to reconstruct complete data from corrupted inputs, converting the previously problematic noisy data into a useful training signal that improves model robustness and accuracy while maintaining fast training convergence.
Solution Approach 2:
The patent applies preliminary data corruption to the training dataset before model training begins. By pre-introducing missing patterns and noise according to learned distributions, the model is prepared in advance to handle real-world incomplete data, eliminating the need for data deletion and enabling direct training on all available data points.
2Reliability
If denoising autoencoders are trained using traditional data duplication methods to handle missing values, then model robustness improves, but processing power consumption and training time increase significantly
Solution Approach 1:
The patent changes the fundamental training parameters by learning the distribution of missing patterns from the data and using this distribution to generate training samples. Instead of duplicating entire datasets, the method varies the parameters of data corruption (missing patterns, noise levels) according to learned distributions, achieving robustness with minimal computational overhead and fast convergence.
3Productivity
If statistical methods are used to impute missing values, then processing speed is maintained, but accuracy deteriorates due to inability to handle outliers and complex data relationships
Solution Approach 1:
The patent replaces traditional statistical imputation mechanisms (mean, median, mode) with a neural network-based denoising autoencoder. This substitution enables the model to learn complex non-linear relationships and handle outliers effectively while maintaining efficient processing speeds through the differentiable nature of neural network operations and batch processing capabilities.
4Adaptability or versatility
If machine learning models are trained to handle noisy and incomplete datasets, then real-world applicability improves, but training complexity and computational resources increase
Solution Approach 1:
The patent implements a self-service approach where the model automatically learns the distribution of missing patterns from the training data itself. The system autonomously identifies data corruption patterns, generates appropriate training samples, and adapts to different data types and missingness mechanisms without requiring manual configuration or complex preprocessing pipelines, thereby reducing training complexity while improving real-world applicability.
Data Source
AI summary
A computing system may include a processor and a memory having a set of instructions, which when executed by the processor, cause the computing system to execute actions. The actions include identifying an estimate of a distribution of missing block patterns, generating a noisy dataset by removing first data from an original dataset based on the estimate and training a denoising autoencoders (DAE) based on the noisy dataset.