Real-Time Data Lineage Extraction via Execution Plan Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data lineage solutions fail to provide complete and accurate tracking of data transformations due to reliance on parsing software code at rest, lack of knowledge about programming languages, and requirement for software architecture information, leading to incomplete data lineage and resource inefficiency.
Innovation Solution
A data lineage tool that works in conjunction with an execution engine to generate an execution plan at runtime, allowing for real-time extraction of complete data lineage by tracing operations from data storage to data extraction, without needing knowledge of programming languages or software architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If existing solutions parse software code at rest to extract data lineage, then the process can be performed without runtime execution, but the data lineage generated is incomplete because not all parameter values are known prior to runtime
Solution Approach 1:
The system performs preliminary code analysis and instrumentation during the build or deployment phase, inserting metadata collection hooks into the code without executing it. This preliminary action prepares the code structure for complete lineage tracking while allowing runtime execution to provide the missing dynamic information.
Solution Approach 2:
The system implements a feedback mechanism where the execution engine's runtime behavior is monitored and fed back to the data lineage extraction tool. This feedback loop allows the system to capture actual parameter values and dynamic code paths during execution, completing the lineage information that static analysis could not obtain.
2Adaptability or versatility
If existing solutions parse software code written in multiple programming languages, then the system can handle diverse applications, but knowledge of these programming languages is required
Solution Approach 1:
The system introduces an intermediary layer consisting of language-agnostic metadata schemas and standardized execution interfaces. This intermediary translates diverse programming language constructs into a unified representation that the data lineage tool can process without requiring language-specific knowledge, while still maintaining the ability to handle multiple languages.
Solution Approach 2:
The execution engine implements universal instrumentation hooks and metadata collection mechanisms that work across multiple programming languages through a common interface. This universal approach allows the same data lineage extraction process to function with different languages without requiring language-specific parsers or analysis tools.
3Loss of information
If existing solutions require information about software architecture to generate end-to-end data lineage, then the lineage can be comprehensive, but the process becomes complex and requires detailed architectural knowledge
Solution Approach 1:
The system implements self-service automation where the execution engine automatically discovers and reports its own architectural components, data flows, and transformation logic during runtime. This eliminates the need for external tools to manually analyze or know the software architecture, as the system generates its own lineage information through automated metadata collection from execution traces.
Solution Approach 2:
The execution engine provides continuous feedback about its operational state, including component interactions and data transformations, to the data lineage extraction tool. This feedback mechanism allows the system to automatically build a complete picture of the software architecture and data flows without requiring external architectural knowledge or manual configuration.
Data Source
AI summary
A system includes an execution engine and a processor. The execution engine receives application code that, when executed, is configured to generate a second data element from a first data element that is stored in a first database and store the second element in a second database. The execution engine converts the code into an execution plan and executes it. The execution plan includes a first operation to obtain the first element from the first database, a second operation to apply a transformation to the first element, and a third operation to store the first element as the second element in the second database. The processor accesses the execution plan and determines that the third operation includes storing the second element. In response, the processor generates a data lineage for the second element by extracting, from each operation from the third to the first, a portion of the data lineage.