Bipartite Lineage Graph for Data Access Event Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data lineage analysis systems face challenges in efficiently and accurately tracking the lineage of data sets created and modified by multiple software processes, particularly in large-scale systems, and in providing timely and accurate lineage data to support auditing and security measures.
Innovation Solution
The method involves generating a bipartite lineage graph from data access event information, which includes timestamp data, to represent software processes and data sets, allowing for efficient traversal and identification of lineage data in response to requests, thereby enabling effective tracking and auditing of data access and propagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data lineage analysis methods are used to track data in large-scale systems, then comprehensive lineage tracking is achieved, but system complexity and processing time increase significantly
Solution Approach 1:
The patent introduces an intermediary component that captures data access events and stores them in a standardized format. This intermediary layer decouples the complexity of lineage tracking from the actual data access operations, allowing comprehensive tracking without proportionally increasing system complexity. The intermediary event capture mechanism serves as a buffer that simplifies the overall system architecture.
Solution Approach 2:
The patent segments the lineage tracking system into distinct components: event capture modules, event storage, and query processing. By dividing the monolithic lineage tracking function into smaller, independent segments, the system can handle large-scale data tracking while maintaining manageable complexity in each individual component.
2Measurement precision
If detailed data access events are tracked for all software processes, then accurate lineage information is obtained, but data processing time and storage requirements increase
Solution Approach 1:
The patent implements preliminary action by capturing and storing data access events as they occur, before lineage queries are submitted. This advance capture and storage of event data eliminates the need for real-time analysis during query processing, significantly reducing query response time while maintaining complete and accurate lineage information.
Solution Approach 2:
The patent creates copies of data access events in a standardized format for storage and analysis, rather than directly querying the actual data access mechanisms. This copying approach allows efficient retrieval and analysis of lineage information without imposing overhead on the original data processing operations.
3Reliability
If comprehensive data access monitoring is implemented, then unauthorized accesses can be detected, but system performance and operational speed decrease
Solution Approach 1:
The patent replaces direct mechanical monitoring of data access operations with an event-based capture system. Instead of continuously monitoring and analyzing data flows in real-time, the system captures access events discretely and stores them for later analysis, reducing the performance overhead while maintaining comprehensive security auditing capability.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for analyzing lineage data. One of the methods includes obtaining data access event information; generating a lineage graph from the data access event information; receiving a data lineage request, wherein the data lineage request is a request for lineage data for a specified data set or a specified software process; traversing the lineage graph to identify nodes and edges in the lineage graph that satisfy the data lineage request; and providing data identifying the nodes and edges that satisfy the data lineage request in response to the request.