Correlation-Based Element-Level Data Lineage for Large Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data lineage records in enterprise databases are difficult to record and track due to the numerous processes, data elements, and rapid transformations, leading to inefficiencies and incomplete lineage graphs in existing solutions.
Innovation Solution
A computer-based system with a lineage module that generates element-level data lineage mapping using a correlation model, determining correlations between interconnected data elements and graphing these relationships without requiring real-time resource consumption or transformations, utilizing processors, memory, and storage to accurately predict element-level mappings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data lineage tracking methods are used to record all processes and data transformations, then complete lineage records can be maintained, but the system complexity and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the essential correlation information between data elements from the complex web of all processes and transformations. Instead of recording every process detail, the system identifies and stores only the statistical correlations between input and output data elements, significantly reducing the amount of data to be tracked while maintaining lineage accuracy.
Solution Approach 2:
The system creates a simplified representation of data lineage through correlation models that copy the essential relationships between data elements without replicating the entire complex process history. This allows the system to maintain lineage information in a reduced, manageable format that preserves the critical relationships.
2Measurement precision
If real-time tracking of all data transformations is implemented, then accurate element-level mappings can be determined, but resource consumption and processing time increase
Solution Approach 1:
The system performs correlation analysis in advance during data processing, establishing statistical relationships between data elements before they are needed for lineage determination. This preliminary computation of correlations allows for accurate element-level mappings without requiring continuous real-time processing, reducing overall resource consumption.
Solution Approach 2:
The patent changes the approach from tracking every transformation detail to measuring statistical correlation parameters between data elements. By using correlation coefficients and statistical relationships as the tracking mechanism, the system achieves precise element-level mappings with significantly reduced processing requirements compared to traditional real-time tracking methods.
3Loss of information
If comprehensive data lineage records are maintained for all applications and processes, then complete data flow understanding is achieved, but the difficulty of recording and tracking increases
Solution Approach 1:
The patent introduces correlation models as intermediary representations that mediate between the complex reality of data transformations and the simplified need for lineage tracking. These correlation models capture the essential data flow relationships without requiring direct tracking of every process detail, making the recording and tracking operationally much easier while preserving complete data flow information.
Data Source
AI summary
In some embodiments, the present disclosure provides an exemplary technically improved computer-based method utilizing an element level mapping module that determines a correlative relationships of interconnected nodes and edges in relation to an output data element and an input data element, and determines an appropriate graph of the relationships.