Variational Autoencoder Data Augmentation for ML Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data cleaning techniques for machine learning applications face challenges in efficiently addressing noisy data, particularly due to the complexity of error types and the need for domain knowledge, leading to suboptimal performance and increased computational complexity.
Innovation Solution
The approach involves data augmentation using a variational autoencoder to increase the amount of clean data, reducing the impact of noisy records by generating synthetic data from the clean fraction of the dataset, thereby improving the quality of data for machine learning models without the need for complex repair candidate selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data cleaning techniques are used to address noisy data, then data quality can be improved, but computational complexity and time consumption increase significantly
Solution Approach 1:
The patent applies the copying principle by creating synthetic copies of clean data samples through the variational autoencoder. Instead of manually cleaning and repairing each noisy record, the system generates artificial clean data copies that replicate the underlying data distribution, thereby improving data quality without the computational complexity of traditional repair methods
Solution Approach 2:
The patent extracts the clean fraction from the dirty dataset by identifying and separating clean records from noisy ones. This extraction is performed using error detection mechanisms that flag erroneous records, allowing the system to isolate and work only with the clean subset for training the variational autoencoder, thus reducing computational requirements compared to processing the entire dataset
2Reliability
If data repair techniques are applied to fix erroneous samples, then the proportion of clean data increases, but the process requires domain knowledge and manual intervention
Solution Approach 1:
The patent implements self-service by enabling the system to automatically detect errors, extract clean data, and generate synthetic clean samples through the variational autoencoder. The error detection mechanisms and automated sampling processes eliminate the need for manual domain knowledge and intervention, allowing the system to self-clean and self-augment the dataset
Solution Approach 2:
The system creates synthetic copies of clean data samples through the variational autoencoder, replacing the need for manual repair techniques. These synthetic copies are generated by learning the underlying data distribution from the clean fraction and applying it to generate new clean samples, thereby increasing the proportion of clean data without requiring manual domain expertise
3Reliability
If the entire dataset is processed to detect and repair errors, then data quality improves, but the time required for data preparation increases
Solution Approach 1:
The patent extracts only the clean fraction from the dirty dataset, avoiding the time-consuming process of processing and repairing every record. By identifying and isolating clean records, the system can efficiently train the variational autoencoder on a smaller, high-quality subset, significantly reducing data preparation time while maintaining data quality
Solution Approach 2:
The system generates synthetic copies of clean data samples through the variational autoencoder, which is much faster than manually processing and repairing every erroneous record. This copying approach allows the system to quickly expand the clean data proportion without the time cost of thorough manual cleaning and repair operations
Data Source
AI summary
Certain example embodiments relate to automated data preparation techniques usable to improve machine learning (ML) pipelines. Different error detectors are executed on a dirty dataset to identify which records therein include errors. Each record that has been identified by at least a threshold number of the error detectors as including an error is marked as erroneous. The dirty dataset's records are divided into clean and dirty fractions. If a data exclusion error has emerged, the process is repeated. Otherwise, a new set of data samples is generated by applying a variational autoencoder (VAE) to the clean fraction, the dirty dataset is augmented with the new set of data samples, and the augmented dirty dataset is provided for training of the ML model. It thus becomes possible to better training an ML model without having to repair data determined to include errors.