Synthetic Data Generation Using Clustered Categorical Variables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data simulation techniques face challenges with exponentially increasing contingency table sizes due to numerous categorical variables and categories, leading to memory and processing issues that hinder efficient data generation.
Innovation Solution
The method involves clustering categorical variables based on their associations, identifying a key variable for each cluster, and generating cluster contingency tables, which are then used to create an artificial data set by joining the cluster data sets using the key cluster as a join key.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a contingency table is created to include all combinations of categorical variables, then the data generation accuracy is improved, but the memory and processing requirements increase exponentially
Solution Approach 1:
The patent divides the categorical variables into multiple clusters, where each cluster contains a subset of variables. Instead of creating one large contingency table with all possible combinations, the system creates separate contingency tables for each cluster. This segmentation reduces the memory and processing requirements while maintaining the ability to generate accurate synthetic data by combining results from individual cluster tables.
2Adaptability or versatility
If the number of categorical variables and categories increases, then the data simulation comprehensiveness is improved, but the contingency table size increases exponentially making it unusable
Solution Approach 1:
The system segments the large set of categorical variables into multiple smaller clusters. Each cluster is processed independently to create a manageable contingency table, avoiding the exponential size explosion that would occur with a single comprehensive table. The key variables from each cluster are then used to join the results, maintaining comprehensiveness across all variables.
Solution Approach 2:
The patent introduces key variables as intermediaries between clusters. These key variables are selected from each cluster and used to join the synthetic data generated from different cluster contingency tables. This intermediary approach allows the system to maintain relationships across all categorical variables without requiring a single large contingency table containing all possible combinations.
Data Source
AI summary
A computer-implemented method for generating an artificial data set is provided. Aspects include obtaining an input data set, calculating an association between the plurality of categorical variables of the input data set, and creating, based on the association, a plurality of clusters of categorical variables. Aspects also include identifying a key variable for each of the plurality of clusters of categorical variables, creating a key cluster for each of the plurality of clusters, and creating a cluster contingency table for each of the clusters. Aspects further include generating, based on the cluster contingency table for each of the plurality of clusters and for the key cluster, a data set for each of the plurality of clusters and the key cluster and generating the artificial data set based on a combination of the data set for each of the plurality of clusters and the key cluster.