Catalog-Based Data Revision Control for Branch Dataset Builds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional large-scale data analytic systems lack effective revision control for data processed within them, limiting their functionality in managing data and job results.
Innovation Solution
Implement a data revision control system that records and tracks transformations of data through a catalog service, providing versioned immutable datasets, dataset provenance, and build dependency management, ensuring data integrity and permission control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional large-scale data analytic systems store data in distributed file system locations without revision control, then data processing efficiency is maintained, but data revision control capability is lost
Solution Approach 1:
The system segments data management into distinct components: a catalog service that handles revision control metadata and a distributed file system that stores actual data. This segmentation allows the data processing system to maintain efficiency while the catalog service provides specialized revision control functionality through version identifiers and provenance tracking.
Solution Approach 2:
The catalog service acts as an intermediary between data processing jobs and the distributed file system. It mediates by recording provenance information, managing version identifiers, and tracking build dependencies without interfering with the efficient data processing operations in the distributed file system.
2Reliability
If a catalog service is implemented to track data provenance and versions, then data revision control is improved, but system complexity increases
Solution Approach 1:
The revision control functionality is extracted from the core data processing system and placed in a separate catalog service. This extraction reduces the complexity of the main data processing system while maintaining reliable revision control through the specialized catalog service that handles version identifiers, provenance tracking, and build dependency management.
Solution Approach 2:
The catalog service provides multiple functions within a single system: storing version identifiers, tracking data provenance, managing build dependencies, and controlling access permissions. This multi-functionality reduces overall system complexity compared to having separate mechanisms for each revision control aspect.
3Reliability
If versioned immutable datasets are maintained with full provenance tracking, then data integrity is ensured, but storage requirements increase
Solution Approach 1:
Instead of storing multiple full copies of datasets for each version, the system creates lightweight copies in the form of version identifiers and provenance metadata in the catalog service. The actual data remains stored once in the distributed file system, while the catalog service maintains references and version information that enable immutable dataset versioning without duplicating the large data volumes.
Data Source
AI summary
A method comprises storing, in a build catalog, for each update of a dataset, an entry including a branch identifier, an identifier and a version of the dataset, and build dependency information; receiving a first request to build a second branch having a first branch as a parent branch, the first branch being associated with a first version of a first driver program for building a first dataset from a set of child datasets, the second branch being associated with a second version of the first driver program; determining that the second branch does not have any version of a specific child dataset based on the build catalog; retrieving a latest version of the specific child dataset from the first branch; causing a build of the first dataset based on the latest version of the specific child dataset and the second version of the first driver program.