Time-Aware Data Lineage Graphs for Cross-Application Activity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data lineage systems are non-time aware, leading to inefficient and costly manual processes for maintaining enterprise data, as compiled data lacks time relevance and utility, making large-scale data lineage difficult and time-consuming.
Innovation Solution
A system and method for reconstructing time-aware data activity across multiple software applications by generating time period path graphs, determining temporal centrality of nodes, and comparing communication pathways to identify central nodes and optimize data flow.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If compiled network communication data is used to map data lineage, then data coverage across multiple applications is achieved, but time relevance and utility of the compiled data are lost
Solution Approach 1:
The patent segments the compiled network communication data into discrete time intervals, creating time-aware snapshots of data lineage at different moments. This segmentation allows the system to preserve time relevance information while maintaining comprehensive data coverage across multiple applications, resolving the contradiction between broad coverage and temporal accuracy.
2Ease of operation
If manual processes are used to maintain enterprise data lineage, then flexibility and control are maintained, but time consumption and cost increase significantly
Solution Approach 1:
The patent implements automated detection and reconstruction of data lineage using network communication data and machine learning algorithms. The system self-services by automatically identifying data flows, determining temporal centrality of nodes, and generating time-aware path graphs without requiring manual intervention, thereby reducing time consumption while maintaining operational flexibility through configurable detection parameters.
3Measurement precision
If time-aware path graphs are generated for multiple time periods, then accuracy of data lineage analysis is improved, but system complexity and processing requirements increase
Solution Approach 1:
The patent generates dynamic time-aware path graphs that evolve over multiple time periods, allowing the system to capture temporal dynamics of data lineage. The system maintains manageable complexity by processing data in sequential time intervals and using algorithms that efficiently handle temporal centrality calculations, achieving high analysis accuracy without proportionally increasing system complexity.
Data Source
AI summary
Systems, computer program products, and methods are described herein for reconstructing time aware data activity across multiple software applications. The method includes receiving one or more node communication data packets. Each of the one or more node communication data packets are time-stamped and include two nodes of a plurality of nodes within a network. The method also includes generating a first time period path graph. The first time period path graph includes a directional data flow between the plurality of nodes during a first time period. The first time period path graph is generated based on the one or more node communication data packets. The method further includes determining a central node of the plurality of nodes for the first time period. The central node of the plurality of nodes is determined based on the first time period path graph.