Synthetic Test Data Generation via Statistical Property Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating test data for software applications often rely on organic data, which can contain private identifying entities, posing privacy risks and legal concerns, and manual generation of synthetic data is time-consuming and not scalable.
Innovation Solution
An automated system that receives an organic dataset, identifies private data entities, determines their statistical properties, and generates synthetic data entities matching the data type and properties, using a domain knowledge base to create synthetic test data templates, thereby eliminating the need for manual rule creation and ensuring data privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If organic data is used for testing, then test data effectiveness is improved, but privacy risks and legal concerns increase
Solution Approach 1:
The patent creates synthetic test data that copies the statistical properties, data types, and structural characteristics of organic data without replicating actual private information. This allows effective testing while eliminating privacy risks by using artificially generated data that mirrors the patterns of real data.
Solution Approach 2:
The patent extracts and removes private data entities from the organic dataset, separating the useful statistical properties from the harmful private information. This extraction process allows the system to retain test effectiveness while eliminating privacy concerns.
2Object-affected harmful factors
If manual generation of synthetic data is performed, then data privacy is protected, but time consumption and scalability worsen
Solution Approach 1:
The patent implements an automated system that performs synthetic data generation without manual intervention. The system automatically identifies private data entities, determines statistical properties, generates synthetic data entities, and populates templates, eliminating the time-consuming manual processes while maintaining privacy protection.
Solution Approach 2:
The patent transforms the data generation process from manual parameter specification to automated statistical analysis. By automatically determining statistical properties from the organic data and using these parameters to guide synthetic data generation, the system achieves both privacy protection and high productivity.
3Manufacturing precision
If manual rule creation is required for synthetic data generation, then data accuracy can be controlled, but device complexity and time consumption increase
Solution Approach 1:
The patent replaces manual rule creation with automated statistical analysis and machine learning-based generation. The system automatically learns the patterns, distributions, and relationships in the organic data, eliminating the need for manual rule specification while maintaining or improving data accuracy.
Solution Approach 2:
The patent performs preliminary statistical analysis of the organic data to automatically determine the properties that should be preserved in synthetic data. This preliminary characterization of data patterns enables accurate synthetic data generation without requiring manual rule creation for each testing scenario.
Data Source
AI summary
Embodiments include a method for one or more processors to receive an organic dataset and a domain knowledge base. The one or more processors identify private data entities present within the organic dataset. The one or more processors determine statistical properties of the private data entities identified within the organic dataset. The one or more processors create a plurality of test data templates by removing the private data entities from the organic dataset. The one or more processors select from the domain knowledge base, synthetic data entities that match a data type of the removed private data entities, respectively, and align with the statistical properties of the private data entities, and the one or more processors generate synthetic test data by inserting, respectively, the synthetic data entities of the matching data type for the removed private data entities in the test data templates.