Metadata Management for Data Lineage Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems lack efficient methods for comparing and maintaining multiple versions of data processing systems, particularly in tracing data lineage and assessing system-wide impacts of changes.
Innovation Solution
The system receives user input to select nodes in graphs representing data processing systems, retrieves data lineages for objects associated with these nodes, and generates reports comparing these lineages to identify differences and assess impacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple versions of data processing systems are maintained for comparison and analysis, then the ability to perform impact analysis and trace data lineage is improved, but the system complexity and resource requirements increase
Solution Approach 1:
The patent creates virtual copies of data processing system versions through metadata snapshots. Each version is represented by metadata containing structural information about datasets, transforms, and their relationships. These metadata copies enable comparison and impact analysis without duplicating the actual heavy data processing components, thus maintaining reliability while controlling complexity.
Solution Approach 2:
The patent introduces a metadata layer as an intermediary between the actual data processing systems. This metadata acts as a lightweight representation that captures the essential structure and relationships of multiple system versions. By operating on metadata rather than direct system copies, the patent enables version comparison and impact analysis with reduced complexity.
2Loss of information
If comprehensive data lineage tracing is implemented across multiple data sources and transforms, then the understanding of data relationships is improved, but the time and computational resources required increase
Solution Approach 1:
The patent performs preliminary action by capturing metadata snapshots of data processing systems at specific versions before changes occur. These snapshots pre-record the structural relationships, data sources, and transforms. When impact analysis is needed, the system compares these pre-captured metadata states rather than tracing through actual data flows, dramatically reducing the time required while preserving complete relationship information.
3Measurement precision
If detailed metadata comparison between system versions is performed, then the precision of change detection is improved, but the computational resources and processing time increase
Solution Approach 1:
The patent extracts only the essential structural metadata from complete data processing systems - specifically the schemas, relationships, and transformation definitions - while leaving out the actual data volumes and heavy computational artifacts. This extracted metadata is sufficient for precise change detection in data relationships but requires minimal computational resources to store and compare.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for metadata management. One of the methods includes receiving user input selecting a first node. The method includes receiving a first data lineage of a first object, the first object having a type, the first data lineage describing relationships between the first object and one or more datasets or transforms. The method includes receiving user input selecting a second node. The method includes receiving a second data lineage of a second object, the second object having the same type as the first object. The method includes performing a comparison of the first node and the first data lineage to the second node and the second data lineage. The method includes generating a report based on the comparison.