Synthetic Test Data Generation via Statistical Profiling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Organizations face challenges in generating realistic test data for development and testing purposes without exposing confidential production data, as existing methods either require access to actual data or fail to replicate essential characteristics effectively.
Innovation Solution
A computer-implemented method and system that uses data profiling to extract statistical properties and common values from production data, generating test data that replicates these characteristics while ensuring confidential information is not included, utilizing a data profiling module and test data generation module to create a system for developing programs securely.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If actual production data is used for testing, then test accuracy and realism are improved, but data security and confidentiality are compromised
Solution Approach 1:
The patent creates synthetic test data that copies the statistical properties and patterns of production data without copying actual confidential values. The system generates test data that mimics the distribution, relationships, and characteristics of real data while ensuring no sensitive information is exposed.
Solution Approach 2:
The system extracts only the statistical properties and patterns from production data through profiling, separating the useful structural information from the confidential actual values. This allows the test data to capture the essence of production data patterns without retaining any sensitive information.
2Object-affected harmful factors
If synthetic test data is generated from production data, then data security is improved, but test realism and statistical accuracy may deteriorate
Solution Approach 1:
The system changes the values while maintaining the statistical parameters and distribution characteristics of the original data. By adjusting individual values but preserving the overall statistical properties such as mean, variance, and relationships between fields, the generated data remains statistically accurate and realistic.
Solution Approach 2:
The system performs preliminary data profiling to analyze and understand the statistical properties, relationships, and patterns in production data before generating synthetic test data. This preliminary analysis ensures that the generated data accurately reflects the original data's characteristics.
3Measurement precision
If detailed data profiling is performed to capture all statistical properties, then test data accuracy is improved, but processing time and system complexity increase
Solution Approach 1:
The system performs partial data profiling, focusing on extracting the most critical statistical properties and patterns necessary for generating realistic test data. By selectively profiling key fields and relationships rather than analyzing every aspect of the data in extreme detail, the system achieves sufficient accuracy without excessive processing time.
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
Generating test data includes: reading values occurring in at least one field of multiple records from a data source; storing profile information including statistics characterizing the values; generating a model of a probability distribution for the field based on the statistics; generating multiple test data values using the generated model such that a frequency at which a given value occurs in the test data values corresponds to a probability assigned to that given value by the model; and storing a collection of test data including the test data values in a data storage system.