Flat File Data Migration Referential Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to maintain referential integrity during data migration and masking processes from flat files to relational databases or between flat files, leading to data inconsistencies and loss of meaningful information.
Innovation Solution
A method that defines referential integrity between flat files and relational databases by identifying matching column names and embedded business rules, allowing for user-defined configurations, and ensuring data integrity through automatic detection and migration, while also providing data masking and encryption capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is migrated from flat files to relational databases or between flat files, then data can be made more meaningful and useful, but referential integrity is lost leading to data inconsistencies
Solution Approach 1:
The system performs preliminary analysis of the flat file structure and data relationships before migration begins. It automatically detects potential referential integrity relationships by examining column names, data patterns, and structural characteristics, preparing integrity preservation rules in advance to guide the migration process and ensure consistency is maintained from the outset.
Solution Approach 2:
The system introduces an intermediary layer that acts as a bridge between the source flat files and destination relational databases. This intermediary component analyzes the data, identifies referential relationships, and transforms the data while preserving integrity constraints, effectively mediating the migration process to maintain reliability across different data formats and structures.
2Object-affected harmful factors
If data masking is applied to protect sensitive information during migration, then data security is improved, but data relationships and meaning may be compromised
Solution Approach 1:
The system applies data masking selectively based on local quality requirements. It identifies specific columns and data elements that require masking based on sensitivity classifications, applying masking only to those specific locations while leaving other data elements unchanged. This ensures security is enhanced where needed without compromising the overall meaning and relationships of the dataset.
Solution Approach 2:
The system changes parameters of data elements dynamically during migration. It modifies data representation parameters (such as masking certain digits or characters) based on sensitivity levels, while preserving structural parameters and relationships. This allows the data to maintain its meaningful relationships and integrity while security parameters are adjusted to protect sensitive information.
Data Source
AI summary
A computerized system maintains referential integrity during data migration and optionally masking. The system allows the user to identify one or more flat file(s) as the data source as well as identify the data destination file(s). The system automatically identifies the referential integrity columns where column names match in different flat files, automatically identifies referential integrity by analyzing related application source code for embedded business rules that indicate referential integrity, or can allow the user to describe the referential integrity pertaining to the data source to establish user-defined referential integrity, which referential integrity definition(s) are stored in a data store (e.g., an XML File) accessible via a configuration engine. The system, through a sequence engine and subsequently an execution engine, uses the stored referential integrity and other information to migrate and optionally masks the data from the data source file(s) into the data destination file(s).