Data Lineage Summarization via Graph Node Collapsing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data processing systems become increasingly complex, data lineage diagrams become difficult to understand due to the large number of data nodes presented, making it challenging for users to identify and focus on high-interest data nodes while suppressing low-interest nodes.
Innovation Solution
A method is introduced to manage lineage information by generating summary objects based on paths in a directed graph, collapsing sets of low-interest data lineage nodes into summary nodes represented by a single icon, and displaying only high-interest nodes in detail, thereby reducing complexity and improving understanding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all data nodes are displayed in the data lineage diagram, then completeness of information is improved, but diagram complexity and difficulty of understanding worsen
Solution Approach 1:
The patent segments the data lineage diagram into multiple hierarchical levels or views. Summary nodes group multiple detailed data nodes together, allowing users to view high-level summaries first and drill down to detailed views only when needed. This segmentation resolves the contradiction by maintaining information completeness in detailed views while reducing perceived complexity in summary views.
Solution Approach 2:
The patent introduces an additional dimension of abstraction by creating summary nodes that represent groups of detailed nodes. This adds a hierarchical dimension to the diagram, allowing information to be organized from high-level summaries to detailed views, thereby managing complexity while preserving information completeness across different levels.
2Loss of information
If all data nodes are displayed in the data lineage diagram, then information completeness is improved, but ease of operation and focus on high-interest nodes worsens
Solution Approach 1:
The patent applies local quality by allowing different parts of the diagram to have different levels of detail. Summary nodes provide a simplified view for general navigation, while detailed nodes can be expanded when users need to focus on specific high-interest areas. This enables users to easily navigate the overall structure while maintaining the ability to examine details when needed.
Solution Approach 2:
The patent introduces dynamic behavior where the diagram can transition between summarized and detailed views. Users can interactively expand or collapse summary nodes to reveal or hide detailed information, making the diagram adaptable to different user needs and improving ease of operation by allowing dynamic focus adjustment.
3Measurement precision
If detailed information for all nodes is displayed, then measurement precision and detail accuracy are improved, but readability and understandability worsen
Solution Approach 1:
The patent segments information display into hierarchical levels where summary nodes provide high-level readability and detailed nodes provide measurement precision. Users can navigate from readable summaries to precise details as needed, resolving the contradiction between readability and detail accuracy by separating them into different viewing levels.
Data Source
AI summary
An identification of a directed graph is received that includes data transformation nodes that represent computations that transform data elements and one or more data nodes that represent data elements, and includes directed links that represent lineage relationships; and computing summary information based on paths in the directed graph, and storing the summary information in one or more summary objects. The computing includes: receiving designation of interest for a plurality of the nodes of the directed graph; and generating one or more summary objects for remaining nodes not included in the plurality of nodes of interest, a first summary object including summary information based on a first path between a first node of interest and a second node of interest that does include one or more of the remaining nodes and does not include any nodes of interest other than the first and second nodes.