Synthetic Test Data Generation Preserving Referential Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools for testing and benchmarking SQL engines and ETL processes face challenges in generating high volume, quality test data due to masking of sensitive personally identifiable information (PII), which results in non-testable scenarios and loss of meaningful relationships between data entities, hindering development and quality assurance processes.
Innovation Solution
A computer-implemented system that generates test data based on production data patterns by identifying attributes, generating rules to define these patterns, and applying them to create synthetic data that obscures PII, while maintaining referential integrity and protecting sensitive information, using machine learning and metadata to ensure data quality and volume.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If production data is masked with blanks or pre-defined patterns to protect PII, then security risks are reduced, but testability and meaningful relationships between data entities are lost
Solution Approach 1:
The system changes the parameter of data representation by transforming production data into synthetic data that follows the same statistical patterns and relationships while using different actual values. This allows the data to remain testable and meaningful while protecting PII, as the synthetic data maintains the structural characteristics needed for testing without exposing sensitive information.
Solution Approach 2:
The system creates a copy of production data patterns rather than using the actual data. By copying the statistical properties, relationships, and structural patterns from production data to generate synthetic test data, the system preserves testability while avoiding the use of real PII. The synthetic data is a faithful replica of data patterns without containing sensitive information.
2Object-affected harmful factors
If production data is masked with blanks or pre-defined patterns, then PII protection is improved, but quality and high volume test data availability deteriorates
Solution Approach 1:
The system generates synthetic test data by copying patterns from production data, enabling unlimited generation of high-quality test data without consuming actual production data. This approach allows extensive test data creation while maintaining PII protection, as the synthetic data is derived from patterns rather than actual records.
Solution Approach 2:
The system transforms the data representation parameters to create synthetic versions that maintain statistical properties and relationships. This parameter transformation enables the generation of high-volume test data that reflects production conditions while protecting sensitive information through the synthetic nature of the data.
3Object-affected harmful factors
If masked production data is used for testing, then PII security is improved, but development and quality assurance processes slow down
Solution Approach 1:
The system uses copied synthetic data that replicates production patterns without requiring access to actual production data. This enables developers to perform comprehensive testing including stress testing and performance testing without security constraints, significantly improving development efficiency while maintaining PII security through the synthetic nature of the test data.
Solution Approach 2:
The system performs preliminary generation of synthetic test data that is ready for immediate use in testing environments. By pre-generating data that maintains production-like characteristics and relationships, the system eliminates delays associated with data preparation and masking, allowing developers to immediately begin comprehensive testing while PII remains protected.
Data Source
AI summary
The invention relates to implementing a test data tool that generates test data based on production data patterns. According to an embodiment of the present invention, the test data tool comprises: a processor configured to: receive, via the data input, production data from the one or more production environments, the production data comprises personally identifiable information; identify a plurality of attributes from the production data; for each attribute, identify one or more data patterns; generate one or more rules that define the one or more data patterns for each attribute; generate a configuration file based on the one or more rules; apply the configuration file to generate test data in a manner that obscures personally identifiable information existing in the production data; and transmit the test data to a UAT environment.