Data Transform Provenance Graphs for Column-Level Traceability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed database systems lack the ability to effectively track and visualize the provenance of data transformations, making it difficult to determine how errors arise and what effects changes in transformations may have on data columns, and manual column descriptions are often outdated.
Innovation Solution
A system that generates and visualizes metadata describing the provenance of data columns by transforming source code into structured data objects, which are then used to create a visual graph showing transformation relationships, allowing for the tracking of data transformations and their effects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual column descriptions are used in database systems, then implementation is simple, but the descriptions become outdated and unreliable
Solution Approach 1:
The system automatically generates and maintains column provenance metadata by self-tracking transformations through the data pipeline. The metadata is created and updated automatically by the system itself without requiring external manual intervention, ensuring the information remains current and reliable while reducing operational overhead
Solution Approach 2:
The system pre-calculates and stores provenance metadata alongside the data transformations themselves. By capturing the origin and transformation history at the time the metadata is generated, the system ensures accurate tracking of column provenance without requiring later manual updates
2Adaptability or versatility
If complex data pipelines with multiple transformations are implemented, then data processing capability is improved, but tracking the provenance of individual columns becomes difficult
Solution Approach 1:
The system segments the complex data pipeline into discrete transformation operations, each tracked individually with its own metadata. By breaking down the overall transformation process into separate, manageable segments, the system can track the provenance of columns through each specific operation while maintaining the ability to reconstruct the complete transformation history
Solution Approach 2:
The system introduces provenance metadata as an intermediary layer between the data transformations and the final output. This metadata acts as a mediator that captures and preserves information about each transformation operation, enabling traceability of column origins and transformations without interfering with the actual data processing flow
3Loss of information
If visual tracking of data transformations is implemented, then provenance analysis is improved, but system complexity and computational overhead increase
Solution Approach 1:
The system extracts only the essential provenance information needed for tracking column origins and transformations, separating this critical metadata from the complete transformation history. By extracting and storing only the necessary provenance details, the system reduces storage requirements and computational overhead while maintaining the ability to perform effective provenance analysis
Data Source
AI summary
A method of determining and displaying metadata that represents provenance of columns in a data store comprises receiving query data expressed in a human-readable language and describing one or more transformations of data tables of a distributed database system or columns of the data tables; parsing the query data to create a data structure for unresolved transforms in which the data tables are not matched with the columns; based in part upon table schema metadata describing the columns of the data tables, creating an additional data structure for resolved transforms, the additional data structure matching the data tables with the columns and specifying inputs and outputs of each transformation of the one or more transformations; based on information in the additional data structure, creating and causing display of a visual graph of the columns of the data tables and relationships between the columns, relating to the one or more transformations.