Data Lineage Tracking via Workflow Templates and Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing data lineage across multiple systems, data centers, and availability zones is challenging due to the complexity of tracking data transformations and enrichments, which affects data accuracy and trustworthiness.
Innovation Solution
A method and system for generating data lineage by receiving information about data elements affected by changes, creating lineage templates, and generating lineage graphs to track relationships and workflows, using identification, versioning, and referentiality information to ensure data traceability and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data lineage tracking is implemented across multiple systems and data centers, then data traceability and reliability are improved, but system complexity and difficulty of implementation increase
Solution Approach 1:
The patent segments data lineage tracking into modular components: lineage graphs represent individual data elements and their relationships, while workflow templates define standardized transformation patterns. This segmentation allows the system to manage complexity by breaking down the overall tracking mechanism into manageable, reusable units that can be independently maintained and scaled across multiple systems and data centers.
Solution Approach 2:
The patent introduces workflow templates as intermediary artifacts that mediate between data transformation logic and lineage tracking. These templates serve as standardized intermediaries that capture transformation semantics, making the tracking mechanism more manageable and reducing the complexity of implementing lineage across diverse systems by providing a common interface layer.
2Measurement precision
If comprehensive data lineage information is collected and tracked, then data accuracy and trustworthiness are improved, but information processing overhead and storage requirements increase
Solution Approach 1:
The patent creates simplified copies of data transformation relationships through lineage graphs that reference workflow templates. Instead of storing complete transformation logic and all intermediate data states, the system stores compact lineage representations that point to template definitions, significantly reducing storage requirements and processing overhead while maintaining complete traceability of data transformations.
Solution Approach 2:
The patent performs preliminary action by pre-defining workflow templates that capture common data transformation patterns before actual data processing occurs. This allows the system to reuse established transformation definitions across multiple data elements, reducing the information processing overhead by avoiding redundant tracking of identical transformation logic while maintaining accurate lineage records.
3Stability of the object's composition
If real-time monitoring of data transformations is implemented, then data consistency across systems is improved, but computational resources and processing time are consumed
Solution Approach 1:
The patent implements periodic action through workflow templates that define scheduled or triggered execution of data transformations. Instead of continuous monitoring, the system periodically updates lineage information based on workflow execution events, reducing computational resource consumption while maintaining data consistency across systems through structured, event-driven lineage updates.
Data Source
AI summary
In one aspect, an example methodology implementing the disclosed techniques includes, by a computing device, receiving information about a data element affected by a data change and generating a data lineage of the data element, wherein the data lineage indicates relationships between the data element and one or more other data elements, the one or more other data elements indicated in the received information about the data element. The method also includes, by the computing device, generating a lineage template for the data element based on data lineage information, wherein the lineage template defines a workflow for the data element to complete the data change in the data lineage, and generating a lineage graph based on the data lineage information, wherein the lineage graph tracks the data lineage of the data element. The method may further include monitoring performance of the workflow using the lineage graph.