Data Lineage Querying With SQL Extensions and 3D Visualization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing query languages, such as Structured Query Language (SQL), do not support data lineage queries, making it challenging to trace data from its origin through transformations and storage locations, and generated reports can be confusing for users to navigate.
Innovation Solution
Introduce Structured Query Language (SQL) extensions and visualization mechanisms to simplify data lineage querying, allowing for the generation of data lineage reports that include relationship metadata, and display these reports in 3D through virtual or augmented reality, enabling users to navigate and understand data lineage easily.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing query languages like SQL are used, then query execution is straightforward, but data lineage queries cannot be performed directly
Solution Approach 1:
The patent segments the query language by introducing a specific lineage keyword (e.g., 'LINEAGE') that separates data lineage query functionality from standard SQL operations. This allows the existing SQL framework to remain intact while adding specialized lineage tracing capabilities through a dedicated keyword that triggers lineage report generation.
Solution Approach 2:
The patent makes the query language universal by enabling it to perform both standard data retrieval operations and data lineage tracing operations through the same SQL interface. The lineage keyword acts as a multi-function trigger that extends the query language's capability without requiring a separate querying system.
2Loss of information
If data lineage reports are generated using existing methods, then complete lineage information is obtained, but the reports are confusing for users to navigate
Solution Approach 1:
The patent transforms the flat, two-dimensional lineage report into a three-dimensional visualization that displays data fields, datasets, and data processes as interconnected objects in spatial space. This dimensional transformation allows users to navigate lineage relationships through spatial interactions (clicking, hovering, dragging) rather than parsing linear text, making the complete lineage information more intuitive and easier to explore.
Data Source
AI summary
In one embodiment, a device stores, in a database, relationship metadata indicative of an entry in one data field in the database being derived from that of another data field in the database. The device receives a structured database query that includes an identifier for a particular data field and a lineage keyword that requests a data lineage of the particular data field. The device executes the structured database query on the database to generate a data lineage report for the particular data field by tracing the relationship metadata associated with the particular data field to one or more related data fields. The device provides the data lineage report for display. The data lineage report includes information regarding the one or more related data fields.