Virtual Foreign Keys for Synthetic Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating realistic and secure synthetic data that mimics production environments is challenging, especially in preserving relationships and maintaining input-to-output consistency across tables and databases, which is crucial for pre-production environments and machine learning model training.
Innovation Solution
A platform for generating synthetic data that configures virtual foreign key relationships, uses generators for data transformation, and performs privacy scanning to create secure and realistic data, allowing users to specify source and destination databases and apply transformations such as masking, subsetting, and differential privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synthetic data is generated to mimic production environments, then data realism and security are improved, but difficulty in maintaining relationships and consistency across tables increases
Solution Approach 1:
The patent introduces virtual foreign keys as an intermediary mechanism to maintain referential integrity between tables during synthetic data generation. These virtual foreign keys act as mediators that track relationships between parent and child tables without requiring actual database schema modifications, enabling consistent data generation across multiple tables while preserving relationships
Solution Approach 2:
The patent creates a virtual copy of the database schema that includes virtual foreign key constraints. This virtual schema copy allows the system to simulate and enforce referential integrity rules during synthetic data generation without modifying the actual production database structure, enabling realistic data generation while maintaining relationship consistency
2Stability of the object's composition
If virtual foreign key relationships are configured to maintain referential integrity, then data relationship consistency is improved, but system complexity increases
Solution Approach 1:
The virtual foreign key serves as an intermediary layer that enforces referential integrity without requiring direct database schema modifications. The system maintains stability by tracking parent-child relationships through these virtual keys, which mediate between the synthetic data generation process and the requirement for referential integrity
Solution Approach 2:
The patent segments the foreign key enforcement mechanism into separate virtual foreign key objects that can be independently managed. Each virtual foreign key represents a specific parent-child relationship and can be configured and enforced independently, reducing the complexity of managing overall referential integrity across multiple tables
3Object-affected harmful factors
If data transformations such as masking and subsetting are applied, then security and privacy are improved, but data utility for machine learning may be reduced
Solution Approach 1:
The patent applies different transformation qualities to different parts of the data. Virtual foreign keys maintain exact referential integrity for relationship tracking, while other data fields undergo masking or subsetting transformations. This local differentiation allows security improvements in sensitive fields while preserving utility in fields needed for machine learning
Solution Approach 2:
The virtual foreign key acts as an intermediary that preserves exact relationship information while allowing other data to be transformed. This mediator ensures that referential integrity is maintained even when other data fields are masked or subset, thereby preserving data utility for analysis and machine learning
Data Source
AI summary
Generating virtual foreign key relationships includes determining and displaying, in a user interface, a set of candidate foreign keys. It further includes determining and displaying, in the user interface, a set of candidate primary keys. It further includes receiving a request, via the user interface, to generate a virtual foreign key relationship between a selected candidate foreign key and a selected candidate primary key. It further includes in response to receiving the request to generate the virtual foreign key relationship between the selected candidate foreign key and the selected candidate primary key, generating the virtual foreign key relationship at least in part by generating a record for the virtual foreign key relationship.Determining a schema for non-relational data includes receiving a document in a first format. It further includes scanning the document and determining, for a field in the document, a corresponding path of the field and a data type of the field. It further includes updating a hybrid schema based at least in part on the determined path and data type of the field in the document.