Data Transformation System Maintaining Dependencies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data model transformation technologies face challenges in maintaining dependencies during changes and upgrades of operational systems, leading to significant disruptions and costs, as they require redesigning source-to-target mappings, rebuilding ETLs, and developing new reports, which can take months to restore business continuity for reporting and data management.
Innovation Solution
A system and method for performing real-time data model transformations that receive data from source streams, identify target storage areas, and automatically transform data while maintaining dependencies by utilizing relationship metadata and indexes to ensure referential integrity, allowing for immediate access and storage in formats like Apache HBASE and HDFS.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data model transformation methods are used during operational system changes, then source-to-target mappings and ETLs can be rebuilt, but business continuity is disrupted for months and costs increase significantly
Solution Approach 1:
The system performs preliminary actions by maintaining updated dependency information and transformation mappings in advance before system changes occur. When operational systems change, the pre-existing dependency data enables immediate transformation without requiring months of redesign and rebuilding work, thus maintaining business continuity while minimizing restoration time
Solution Approach 2:
The patent introduces an intermediary layer (the data transformation system with dependency tracking) between the operational systems and reporting systems. This intermediary maintains transformation mappings and dependency information independently, allowing reporting to continue during operational system changes without direct impact, thereby ensuring business continuity and reducing restoration time
2Reliability
If data dependencies are maintained during transformation, then referential integrity is preserved, but transformation complexity increases
Solution Approach 1:
The system segments the complexity by separating dependency tracking from the actual data transformation process. Dependency information is captured and stored as distinct metadata, allowing the transformation engine to work with simplified data while maintaining referential integrity through the separate dependency layer, thus preserving integrity without overwhelming transformation complexity
Solution Approach 2:
The patent creates copies of dependency information and relationship metadata that can be processed independently from the actual data transformation. These copied dependency structures enable referential integrity maintenance through simpler lookup and validation operations rather than complex real-time transformation logic
3Speed
If real-time data transformation is implemented, then immediate access to transformed data is achieved, but processing requirements and system resources increase
Solution Approach 1:
The system performs preliminary processing by pre-computing and storing transformation mappings, dependency relationships, and target format specifications before real-time transformation is needed. This advance preparation enables the real-time system to perform only lightweight lookups and data movement operations rather than full transformation computations, achieving fast transformation while moderating processing resource requirements
Data Source
AI summary
A system, method, and computer program product are provided for performing real-time data model transformations. In operation, a data transformation system receives data from one or more source streams of data. The data transformation system identifies one or more target storage areas to store the data. The data transformation system identifies dependencies associated with the data based on relationship metadata associated with the data. Further, the data transformation system automatically transforms the data into a format associated with the one or more target storage areas such that the dependencies associated with the data are maintained. Moreover, the data transformation system stores the transformed data in the one or more target storage areas in the format associated with the one or more target storage areas such that the dependencies associated with the data are maintained.