Field-Level Data Lineage Tracking From Code Repositories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data governance systems lack effective methods to analyze the complete data flow direction, particularly in determining the association relationship between data, also known as data lineage, which is crucial for ensuring data security, accuracy, and usability.
Innovation Solution
A code analysis-based field lineage analysis scheme is employed to create field query databases from code repositories of nodes, enabling the tracking of data flow by querying for associated fields and determining the flow direction of target data types through syntax trees and database queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If code analysis-based field lineage analysis is implemented to track data flow with field granularity, then measurement precision of data flow direction is improved, but device complexity increases due to creating and querying multiple field query databases from code repositories
Solution Approach 1:
The system performs preliminary actions by creating field query databases from code repositories in advance, before actual data flow tracking is needed. This pre-processing allows the system to have ready-to-use databases that contain field-level information from source code, enabling precise data flow tracking without performing complex code analysis in real-time.
Solution Approach 2:
The patent introduces field query databases as intermediary structures between the code repositories and the data flow analysis system. These databases serve as a mediator that stores pre-extracted field information, allowing the system to query data flow directions without directly analyzing source code each time, thus reducing the complexity of real-time analysis.
2Loss of information
If field query databases are created from code repositories of multiple downstream nodes, then completeness of data lineage information is improved, but loss of time increases due to the extensive querying process across multiple databases
Solution Approach 1:
The system pre-creates field query databases from code repositories of all downstream nodes before data flow analysis is needed. This preliminary action ensures that all necessary field-level information is already extracted and stored, eliminating the need for time-consuming code parsing during actual tracking operations.
Solution Approach 2:
The patent prepares field query databases in advance as a cushioning measure, so that when data flow tracking is required, the system can quickly query pre-existing databases rather than performing extensive code analysis. This beforehand preparation cushions against time losses during critical tracking operations.
3Reliability
If comprehensive data flow tracking is implemented across the entire system, then reliability of data governance is improved, but device complexity increases due to the need to analyze complete data flow directions through multiple nodes
Solution Approach 1:
The patent segments the data flow analysis problem into field-level units by creating separate field query databases for different data fields and nodes. Instead of analyzing entire data flows at once, the system breaks down the analysis into manageable field-specific queries, making the complex task of tracking data across multiple nodes more tractable while maintaining comprehensive coverage.
Solution Approach 2:
The field query databases act as intermediary structures that simplify the complexity of comprehensive data flow tracking. Rather than directly analyzing complex inter-node data flows, the system uses these databases as mediators to store and retrieve field-level flow information, reducing the computational complexity while ensuring complete data lineage tracking.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the disclosure provide a method, apparatus, device and computer readable storage medium for data analysis. In the method, based on respective code repositories of a plurality of nodes downstream of a first node, a plurality of respective field query databases are created, wherein data of a target type flows through the first node; a query is made for a field associated with the data of the target type in the plurality of field query databases; and based on a result of the querying, a flow direction of the data of the target type from the first node to the plurality of nodes is determined. In this way, the data of the target type can be effectively tracked from an overall perspective.