Universal Data Lineage Tracking Across Workflow Engines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data lineage tracking systems are limited to specific workflow engines, programming languages, and database structures, failing to track data flows outside of these engines, which restricts their applicability and effectiveness in big data and machine learning workflows.
Innovation Solution
A data lineage tracking system that is workflow engine agnostic, datastore agnostic, and programming language agnostic, capable of capturing and storing inputs and processing steps to produce data artifacts, and generating a directed graph of data artifacts to show their creation lineage, across multiple workflow engines and datastores, allowing for flexible configuration and integration of data lineage information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional data lineage trackers are tied to specific workflow engines, then implementation complexity is reduced, but adaptability and coverage of data flows are limited
Solution Approach 1:
The patent implements a universal data lineage tracker that can monitor multiple workflow engines (Airflow, Prefect, Luigi, Kubeflow, etc.) through a common agent-based architecture. The tracker uses standardized interfaces and configuration files to adapt to different engines without requiring engine-specific implementations, achieving multi-functionality across diverse workflow management systems.
Solution Approach 2:
The patent introduces intermediary components including agents that run within workflow engines, sidecars that intercept data flows, and configuration files that map engine-specific concepts to a unified lineage model. These intermediaries translate between different workflow engine protocols and the universal tracker, enabling adaptability without direct coupling to specific engines.
2Productivity
If manual data lineage tracking is used, then implementation flexibility is maintained, but time consumption and labor requirements increase
Solution Approach 1:
The patent implements self-service automation where the data lineage tracker automatically discovers workflow definitions, extracts data flow information, and generates lineage graphs without human intervention. The system uses configuration files and automated parsing to capture lineage information, eliminating manual tracking efforts while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary actions by pre-configuring templates for common workflow engines, pre-defining data flow patterns, and pre-establishing connection protocols. This allows the system to automatically recognize and track lineage information as workflows execute, rather than requiring post-hoc manual analysis.
3Loss of information
If data lineage tracking is restricted to workflow engine internal flows, then tracking accuracy within the engine is improved, but overall data flow visibility is reduced
Solution Approach 1:
The patent segments the data lineage tracking system into independent modular components: agents for workflow engine integration, sidecars for data flow interception, configuration modules for engine-specific mappings, and a central lineage database. This segmentation allows the system to track both internal workflow flows and external data flows through separate but coordinated mechanisms.
Solution Approach 2:
The patent implements a nested architecture where agents run within workflow engines, sidecars nest within data pipelines, and configuration files nest within engine-specific directories. This nested structure allows the tracker to operate at multiple levels simultaneously, capturing both engine-internal and cross-engine data flows through layered monitoring.
Data Source
AI summary
Systems, apparatuses, and methods are provided herein for tracking data lineage. A method for tracking data lineage comprises detecting, with a data lineage recorder module, an execution of a task of a workflow by a workflow engine on a device, the workflow comprises a plurality of ordered tasks, detecting, with a data lineage proxy, a request sent from the device, selecting, with the data lineage recorder module, request lineage data items associated with the task from a content of the request, associating, in a data lineage database, the request lineage data items with the task, and generating a data lineage graph comprising representations of the plurality of ordered tasks of the workflow including the task and the request lineage data items associated with the task.