Build Service Adaptor for External Dataset Revision Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional large-scale data analytic systems lack effective revision control management for data processed, limiting their functionality and requiring improved systems and methods for data revision control.
Innovation Solution
The method involves a build service that receives an external dataset and its associated adaptor, queries the adaptor for information about the dataset, and inputs the dataset into a build pipeline using the retrieved information, allowing for concurrent execution of data builds if compatible.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional large-scale data analytic systems process external datasets, then data processing capability is improved, but revision control management is insufficient
Solution Approach 1:
The patent introduces an adaptor as an intermediary component between external datasets and the build service. The adaptor translates external dataset formats into the build service's internal representation, enabling the system to process diverse external data sources while maintaining consistent revision control and data management capabilities throughout the processing pipeline.
2Adaptability or versatility
If external datasets are imported into the build service, then data variety is improved, but reformatting requirements increase complexity
Solution Approach 1:
The adaptor serves as a format-translation intermediary that handles the complexity of converting various external dataset formats into the build service's internal representation. This abstraction layer allows the system to accept diverse external data sources without increasing the complexity of the core build service logic.
Solution Approach 2:
The adaptor implements multi-functionality by supporting multiple external dataset formats through a single unified interface. The build service interacts with all external datasets through the standardized adaptor interface, eliminating the need for separate handling logic for each data format.
3Productivity
If multiple data builds are executed concurrently, then processing efficiency is improved, but data integrity risks increase
Solution Approach 1:
The build service performs preliminary actions by determining compatibility between concurrent data builds before execution. It analyzes build definitions to identify potential conflicts over shared datasets and resolves them by sequencing builds or isolating their data operations, enabling safe concurrent execution without compromising data integrity.
Data Source
AI summary
A method of handling a dataset for a build pipeline comprises receiving, from a build service, a query for information related to a dataset being in a first format not managed by the build service for a build pipeline; providing, to the build service, first information related to change in the dataset since a last build performed by the build service in a second format managed by the build service, the first information indicating a version of the dataset used in the last build; transmitting, to the build service, a request for information regarding a new build performed based on the first information; receiving, from the build service, second information regarding the dataset generated from the new build in the second format; storing a new version of the dataset in the first format based on the second information.