Metadata-Driven Data Transformation Logic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional relational databases struggle to handle large volumes of big data, making it difficult to trace the evolution of individual variables and manage data transformations, especially when output data contains personally identifying information, and often result in duplicative data that is hard to detect and prevent.
Innovation Solution
A system and method for managing data transformation and derivation in big data environments, which receives requests to derive output variables, compares proposed logic to existing logic, generates transformation code, schedules its execution, and stores metadata to track the lineage of variables, thereby reducing duplicative data processing and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional relational databases are used to store big data, then data storage is simple, but the system cannot handle the volume and complexity of big data sets
Solution Approach 1:
The patent segments the monolithic database approach into a distributed data lake architecture, dividing data storage and processing across multiple nodes and formats. This allows the system to handle big data volumes by distributing the load rather than concentrating it in a single relational database.
Solution Approach 2:
The patent transitions from traditional two-dimensional relational tables to a multi-dimensional data lake architecture that incorporates various data formats (structured, semi-structured, unstructured), storage locations, and transformation layers, enabling the system to accommodate big data's complexity and volume.
2Productivity
If transformations are hard coded into systems, then execution is efficient, but retroactively determining the evolution of individual variables becomes difficult
Solution Approach 1:
The patent implements a metadata store that continuously captures and stores information about transformations as they execute. This feedback mechanism allows the system to maintain a complete history of variable evolution while transformations run efficiently, enabling both high productivity and full traceability.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the hard-coded transformation logic and the historical record. This metadata store acts as a mediator that captures transformation details without interfering with execution efficiency, preserving the evolution of variables for later analysis.
3Adaptability or versatility
If data is stored in numerous different data storage formats in various locations, then diverse application parameters are serviced, but tracking the lineage of variables becomes difficult and time consuming
Solution Approach 1:
The patent creates a universal metadata store that serves multiple functions: tracking variable lineage, storing transformation logic, recording execution timestamps, and maintaining data provenance across diverse storage formats. This single multi-functional component enables the system to support various application parameters while simplifying lineage tracking.
4Productivity
If duplicative output data is generated, then processing requirements are met, but processing and storage resources are wasted
Solution Approach 1:
The patent performs preliminary actions by storing metadata about output data before actual data generation occurs. The metadata store captures information about what data will be produced, allowing the system to identify and prevent duplicative transformations before they consume processing and storage resources.
Data Source
AI summary
A system may receive a request to derive an output variable from a source variable. The request may include proposed logic to derive the output variable from the source variable. The system may then compare the proposed logic to existing logic to determine the proposed logic is new. In response to the proposed logic being new, the system may generate transformation code configured to execute the proposed logic. The system may further schedule the transformation code for execution at a predetermined time, and then execute the transformation code to generate data for the output variable.