Data Lineage Tracking via Relationship Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large computer systems, tracing the lineage of big data from its source to its consumption is challenging, especially when data from unreliable sources is combined with data from reliable sources, making it difficult to determine data reliability and quality.
Innovation Solution
The system uses relationship tables to track data lineage by chaining metadata from multiple sources, allowing users to trace the data flow and assign quality scores to each data element based on its reliability, thereby providing visibility into the analytics pipeline and addressing data quality issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data from multiple sources is combined in large computer systems, then data analysis capability is improved, but data reliability and quality become difficult to determine
Solution Approach 1:
The patent introduces metadata as an intermediary element that carries reliability information about data sources. This metadata acts as a mediator between the data itself and the analysis system, enabling the system to understand and evaluate the reliability of data without compromising the ability to analyze diverse data from multiple sources.
Solution Approach 2:
The patent segments data reliability information into separate metadata structures that can be independently tracked and evaluated. By separating reliability metadata from the actual data, the system can maintain the versatility of analyzing diverse data sources while systematically tracking and evaluating reliability through the data lineage.
2Measurement precision
If data lineage tracking is implemented to determine data reliability, then data quality assessment is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary actions by capturing and storing reliability metadata at the point of data generation and transformation. This preliminary documentation of data lineage and reliability information eliminates the need for complex real-time analysis during data processing, simplifying the overall system while enabling precise data quality assessment.
Solution Approach 2:
The patent creates copies of reliability information through metadata that replicate data lineage characteristics. These metadata copies preserve the essential reliability attributes without requiring the actual data to be repeatedly processed or analyzed, reducing computational complexity while maintaining measurement precision.
3Loss of information
If metadata chaining is used to track data flow, then data lineage visibility is improved, but processing time increases
Solution Approach 1:
The patent performs the action of capturing and storing lineage information in metadata at the time of data transformation. This preliminary recording ensures that data lineage visibility is maintained without requiring time-consuming retrospective analysis, as all necessary lineage information is already documented and ready for retrieval.
Solution Approach 2:
The patent extracts essential lineage information from the complex data processing operations and places it into separate metadata structures. This extraction allows the system to maintain comprehensive data lineage visibility while separating the lineage tracking function from the main data processing operations, thereby minimizing processing time overhead.
Data Source
AI summary
In an example, one or more processes that transform data originating in one or more of a plurality of different data sources are identified. Then, for each of the identified one or more processes, a relationship table is created, the relationship table listing objects input to the process, objects output from the process, and one or more mappings defining transformations performed on the objects input to the process to produce the objects output from the process. Dependencies are created between relationship tables by linking at least an instance of a first object input to a process in a first relationship table with at least an instance of the first object output from a process in a second relationship table. Lineage of an object in the computer system is traced by accessing one or more of the relationship tables and the dependencies.