Column Lineage and Metadata Propagation in Data Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data pipeline systems fail to automatically propagate column metadata, such as lineage and access control information, from source to target datasets, leading to errors, security breaches, and increased manual effort due to the complexity of transformation code written in various programming languages.
Innovation Solution
The system automatically generates target column metadata by deriving relationships between source and target columns using a logical query plan, which is programming language agnostic, and inferring relationships based on column names, data types, and statistics, thereby including user comments, access control policies, and lineage metadata.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual metadata propagation is used, then metadata accuracy is maintained, but labor intensity and time consumption increase significantly
Solution Approach 1:
The system enables automatic self-propagation of metadata through transformation code analysis. The metadata management system automatically extracts, transforms, and propagates metadata from source to target datasets by analyzing transformation code and deriving column relationships, eliminating the need for manual metadata propagation while maintaining accuracy.
Solution Approach 2:
The patent replaces manual mechanical processes with automated computational analysis. Instead of manually tracking and propagating metadata through complex transformation pipelines, the system uses automated code analysis, logical query plan generation, and relationship derivation to substitute human effort with machine-based metadata propagation.
2Adaptability or versatility
If transformation code in multiple programming languages is analyzed, then comprehensive metadata coverage is achieved, but software complexity increases
Solution Approach 1:
The system implements a universal transformation code analysis framework that handles multiple programming languages through a common architecture. The metadata management system uses language-agnostic logical query plans and standardized relationship derivation mechanisms that work across different programming languages, providing multi-functionality without proportionally increasing complexity.
Solution Approach 2:
The patent changes the analysis parameters from language-specific syntax details to language-agnostic semantic concepts. By focusing on transformation logic, data relationships, and metadata semantics rather than programming language specifics, the system achieves multi-language support while maintaining consistent analysis complexity through parameter abstraction.
3Measurement precision
If column relationships are derived from logical query plans, then metadata propagation accuracy is improved, but processing overhead increases
Solution Approach 1:
The system performs preliminary generation of logical query plans from transformation code before metadata propagation. By pre-computing the logical query plan and column relationships upfront, the system enables accurate metadata derivation without repeating complex analysis during each propagation operation, reducing overall processing overhead while maintaining precision.
Solution Approach 2:
The patent uses copying of logical query plans and relationship models to avoid re-analysis. Once column relationships are derived from transformation code, the system creates reusable representations of these relationships that can be copied and applied across multiple metadata propagation scenarios, reducing processing overhead while maintaining derivation accuracy.
Data Source
AI summary
Whether a logical query plan to derive a target dataset from one or more source datasets is available is determined. Responsive to determining that the logical query plan is not available for the transformation code in the first programming language, relationships between the source columns of the one or more source datasets and the respective target columns of the target dataset are inferred. Target column metadata for a target column of the target dataset is generated. The target column metadata reflects the inferred relationships between the source columns of the one or more source datasets and the respective target columns. The target column metadata for the target column of the target dataset is stored.