VAE Training on Heterogeneous Public Tabular Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Publicly available data for machine learning is often heterogeneous in format and content, making it difficult to train models effectively due to inconsistencies in formatting and consistency, which hinders the efficiency of downstream tasks like regression and classification.
Innovation Solution
A variational auto-encoder (VAE) model is trained on a plurality of heterogeneous data tables to generate synthetic data tables, using mechanisms like Gaussian Mixture Models for normalization, column signature generation, and autoencoders to standardize and generate synthetic data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If publicly available data is used for training machine learning models, then the quantity of training data is increased, but the heterogeneity in format and content reduces training efficiency
Solution Approach 1:
The patent introduces an intermediary processing system that acts as a mediator between heterogeneous public data and machine learning models. This system performs automatic format standardization, data validation, and quality filtering to transform diverse data sources into a unified training format, thereby enabling efficient model training on large quantities of public data without manual preprocessing
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting data processing parameters based on the specific characteristics of each heterogeneous data source. The system automatically detects data formats, schemas, and quality metrics, then transforms these varying parameters into a standardized target format suitable for machine learning training, resolving the conflict between data quantity and training efficiency
2Reliability
If data formatting and consistency issues are addressed manually, then data quality is improved, but the time and resources required increase significantly
Solution Approach 1:
The patent implements self-service by enabling the data processing system to automatically perform formatting standardization, quality validation, and consistency checking without human intervention. The system uses automated schema detection, format inference, and quality metric calculation to handle data preprocessing tasks that would otherwise require manual effort, significantly reducing preprocessing time while maintaining high data quality
Solution Approach 2:
The patent applies preliminary action by performing data format standardization and quality validation as preliminary steps before the main machine learning training process. The system pre-processes heterogeneous data sources to establish consistent formats, validate schemas, and filter low-quality records in advance, thereby eliminating the need for time-consuming manual preprocessing during model training
Data Source
AI summary
A method, computer program product and system are provided for feature engineering and synthetic data generation. A processor retrieves a plurality of data tables, where the plurality of data tables are heterogeneous in format and content. A processor trains a variational auto-encoder (VAE) model on the plurality of data tables. A processor receives an input data table. A processor generates a synthetic data table based on the input data table and the trained VAE model.