Structured Synthetic Data Generation With Privacy-Preserving Realism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative machine-learned models generate repetitive or unrealistic data, fail to replicate realistic production data, and may violate privacy or cybersecurity best practices, and require manual user intervention to tailor test data for software environments.
Innovation Solution
A machine-learned model pipeline generates structured synthetic data by querying metadata for data structures, using transformer-based models to create realistic objects that obfuscate personal information and preserve original characteristics, reducing the need for manual rule-setting and ensuring data conformity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If generative machine-learned models are used to create test data, then data generation speed is improved, but data realism and suitability for testing deteriorates
Solution Approach 1:
The system segments the data generation process into multiple specialized components: a generative model for rapid data creation, a realism evaluation module for quality assessment, and an iterative refinement mechanism. This segmentation allows each component to optimize for its specific function while maintaining overall system reliability.
Solution Approach 2:
The system implements feedback loops where generated data is evaluated against realism criteria, and the generative model is retrained or adjusted based on evaluation results. This closed-loop feedback ensures that rapidly generated data progressively improves in realism while maintaining generation speed.
2Quantity of substance
If general data sets are used for model training, then training data availability is improved, but data specificity to particular software environment deteriorates
Solution Approach 1:
The system performs preliminary action by first training on general datasets to establish baseline capabilities, then systematically fine-tuning with environment-specific data. This staged approach ensures both data availability for initial training and specificity for eventual deployment.
Solution Approach 2:
The system changes training parameters progressively: starting with general data configurations, then adjusting to environment-specific parameters and constraints. This parameter transformation allows the model to adapt from general-purpose to specialized functionality.
3Reliability
If realistic production data is used for testing, then test data realism is improved, but privacy and cybersecurity compliance deteriorates
Solution Approach 1:
The system creates synthetic copies of production data that replicate its statistical properties, relationships, and patterns without containing actual sensitive information. These synthetic copies provide realistic testing conditions while eliminating privacy risks associated with using real production data.
4Adaptability or versatility
If manual user intervention is required to tailor test data, then data adaptability to environment changes is improved, but operation complexity and time consumption deteriorates
Solution Approach 1:
The system implements self-service by automatically detecting environment changes, retrieving updated data structures, and retraining or adjusting the generative model without user intervention. This automation maintains data adaptability while eliminating manual operational complexity.
Solution Approach 2:
The system dynamically adapts to environment changes by continuously monitoring for updates and automatically adjusting its data generation parameters. This dynamic behavior provides adaptability comparable to manual intervention but without the operational burden.
5Manufacturing precision
If users manually define data structures and schemas, then data conformity to environment requirements is improved, but device complexity and time consumption deteriorates
Solution Approach 1:
The system introduces an intermediary component that automatically retrieves data structures and schemas from the software environment and translates them into training configurations for the generative model. This intermediary maintains data conformity while eliminating the complexity and time burden of manual definition.
Data Source
AI summary
Techniques may generate realistic synthetic data by programmatically generating a configuration file object type and relationship data. This configuration file may be used to retrieve source data matching the object type(s) and/or specific records indicated by the configuration file. The techniques may detect and anonymize private/proprietary information and may determine statistical characteristic(s) of the source data. A batch of prompt(s) may be generated using the source data, the statistical characteristic(s), and the configuration file and may be transmitted to one or more instances of a transformer-based machine-learned model. Sets of synthetic data received from the model instance(s) may be de-duplicated, checked for similarity to the source data (e.g., via embedding the synthetic data and the source data), and may be used to generate synthetic object(s) using the relationship(s) and/or other data indicated by the configuration file. These synthetic object(s) may then be deployed in a software environment.