Conditional GAN Autoencoder for Synthetic Tabular Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating synthetic tabular data struggle to accurately preserve pairwise correlations between variables, which is crucial for training machine learning models, especially when dealing with high cardinality and sparse data, and often face challenges related to data privacy and cost.
Innovation Solution
A method utilizing a conditional generative adversarial network (GAN) combined with an autoencoder (AE) to generate synthetic data, where a conditional generator model ensures even sampling of categories and a discriminator model classifies data as real or fake, while the AE preserves correlations in a lower dimensional feature space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing methods are used to generate synthetic tabular data, then data privacy is protected and cost is reduced, but the pairwise correlations between variables are not accurately preserved
Solution Approach 1:
The patent combines a conditional GAN framework with an autoencoder architecture, merging two separate machine learning models into a unified system. The encoder extracts features from real data while the generator creates synthetic samples, and the decoder ensures the synthetic data preserves the original data distribution and pairwise correlations, resolving the contradiction between correlation accuracy and system complexity
Solution Approach 2:
The discriminator model provides feedback to both the generator and encoder by classifying whether input data is real or synthetic. This feedback mechanism guides the training process to improve the quality of synthetic data and ensure accurate preservation of pairwise correlations between variables
2Reliability
If more tabular data is used to train the machine learning model, then the expected model performance is improved, but data capture becomes more difficult and expensive
Solution Approach 1:
The patent creates synthetic copies of real tabular data using the conditional GAN with autoencoder architecture. These synthetic copies preserve the statistical properties and pairwise correlations of the original data, providing sufficient training data for machine learning models without requiring additional real data collection, thereby reducing cost and difficulty of data capture
3Object-affected harmful factors
If synthetic data is generated to improve data privacy, then data privacy is protected, but the statistical realism and pairwise correlations are compromised
Solution Approach 1:
The patent transforms the input data through the encoder into a latent representation, then the generator creates synthetic data in this transformed space while preserving the essential statistical properties. The decoder transforms the synthetic data back to the original space, ensuring that pairwise correlations and statistical realism are maintained while the data remains synthetic and privacy-protecting
Data Source
AI summary
(A) Conditional vectors are defined. (B) Latent observation vectors are generated using a predefined noise distribution function. (C) A forward propagation of a generator model is executed with the conditional vectors and the latent observation vectors as input to generate an output vector. (D) A forward propagation of a decoder model of a trained autoencoder model is executed with the generated output vector as input to generate a plurality of decoded vectors. (E) Transformed observation vectors are selected from transformed data based on the defined plurality of conditional vectors. (F) A forward propagation of a discriminator model is executed with the transformed observation vectors, the conditional vectors, and the decoded vectors as input to predict whether each transformed observation vector and each decoded vector is real or fake. (G) The discriminator and generator models are updated and (A) through (G) are repeated until training is complete.