Immutable Versioned Data Pipeline History Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data pipeline systems require significant manual effort for maintenance and lack dataset versioning, making it difficult to track and preserve data history, especially when updating datasets, which can lead to issues with incorrect data being overwritten and errors in data transformation.
Innovation Solution
A history preserving data pipeline system that maintains immutable and versioned datasets by storing build catalogs and transaction entries, allowing for the tracing of dataset data to its source and derivation program versions, and automating the build process through directed acyclic graphs of build dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data transformation steps overwrite old dataset versions with new versions, then data processing efficiency is improved, but data history tracking capability deteriorates
Solution Approach 1:
The patent creates a copy of the dataset versioning system by maintaining build catalog entries that reference transaction identifiers. Instead of physically storing multiple complete dataset versions, the system creates metadata copies (build catalog entries) that point to the actual data versions through transaction IDs, enabling history tracking without duplicating large data volumes.
Solution Approach 2:
The build catalog serves as an intermediary layer between the data storage system and the versioning requirements. It stores transaction identifiers and build dependency information, acting as a mediator that enables historical data tracking without requiring direct storage of multiple dataset versions, thus resolving the conflict between storage efficiency and versioning capability.
2Adaptability or versatility
If manual configuration is used to maintain data pipeline systems, then system flexibility and adaptability are improved, but maintenance effort and time consumption increase
Solution Approach 1:
The system implements self-service by automatically maintaining build catalog entries and tracking dataset versions through transaction identifiers. The automated build process recursively creates build catalog entries that document data transformations, eliminating the need for manual configuration and reducing maintenance effort while preserving system adaptability.
Solution Approach 2:
The build catalog provides feedback about data lineage and transformation history by storing build dependency information. This automated feedback mechanism tracks which datasets depend on which source data and transformation steps, enabling the system to automatically manage complexity without manual intervention.
3Reliability
If complete dataset versions are stored for history preservation, then data追溯 capability is improved, but storage requirements and system complexity increase
Solution Approach 1:
Instead of copying complete dataset versions, the patent copies only the essential metadata (transaction identifiers and build dependency information) in the build catalog. This lightweight copying approach enables full data追溯 capability by referencing original data versions without duplicating large data volumes, thus reducing storage requirements and system complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A history preserving data pipeline computer system and method. In one aspect, the history preserving data pipeline system provides immutable and versioned datasets. Because datasets are immutable and versioned, the system makes it possible to determine the data in a dataset at a point in time in the past, even if that data is no longer in the current version of the dataset.