Column Provenance Graphs for Distributed Data Transformations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed database systems lack the ability to effectively track and visualize the provenance of data transformations, making it difficult to determine how errors arise and what effects changes in transformations may have on data columns, and manual column descriptions are often outdated.
Innovation Solution
A system that generates and visualizes metadata describing the provenance of data columns by transforming source code into structured data objects, which are then used to create a visual graph showing transformation relationships, allowing for the tracking of data transformations and their impacts on columns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If manual commenting is used to describe columns, then initial documentation is provided, but the descriptions quickly become outdated and inaccurate
Solution Approach 1:
The system automatically generates and maintains column provenance metadata by tracing data transformations through the pipeline itself, without requiring manual intervention. The provenance information is self-updated as transformations occur, ensuring accuracy without human effort.
Solution Approach 2:
The system pre-calculates and stores provenance metadata as transformations are applied to data columns. By capturing transformation information at the time of execution rather than retrospectively, the system ensures documentation is always current without requiring post-hoc updates.
2Adaptability or versatility
If complex data pipelines with multiple transformations are implemented, then data processing capability is enhanced, but tracking the provenance of individual columns becomes intractable
Solution Approach 1:
The system breaks down complex data pipelines into individual transformation operations, tracking provenance at the level of specific column transformations. Each transformation is recorded as a discrete event with clear input-output relationships, making tracking manageable even in complex pipelines.
Solution Approach 2:
The system introduces provenance metadata as an intermediary layer between data transformations and users. This metadata acts as a mediator that captures transformation information without interfering with the actual data processing operations, enabling tracking without adding operational complexity.
3Loss of information
If detailed provenance tracking is implemented for all columns, then complete transformation history is available, but system overhead and computational cost increase
Solution Approach 1:
The system applies provenance tracking selectively to specific columns that require monitoring, rather than uniformly to all data. Users can specify which columns need provenance tracking based on their importance, reducing computational overhead while maintaining complete history for critical columns.
Solution Approach 2:
The system implements provenance tracking at the appropriate level of detail - capturing essential transformation information without recording every minor operation. This partial action approach provides sufficient provenance information for most use cases without the full computational cost of exhaustive tracking.
Data Source
AI summary
A method of determining and displaying metadata that represents provenance of columns in a data store comprises receiving query data expressed in a human-readable language and describing one or more transformations of data tables of a distributed database system or columns of the data tables; parsing the query data to create a data structure for unresolved transforms in which the data tables are not matched with the columns; based in part upon table schema metadata describing the columns of the data tables, creating an additional data structure for resolved transforms, the additional data structure matching the data tables with the columns and specifying inputs and outputs of each transformation of the one or more transformations; based on information in the additional data structure, creating and causing display of a visual graph of the columns of the data tables and relationships between the columns, relating to the one or more transformations.