Branch-Based Dataset Synchronization for Atomic Time-Series Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Ensuring that time-sensitive datasets are up-to-date across different entities in an organization, particularly for mission-critical tasks, is laborious and requires numerous data messages and checks, which can be inefficient and error-prone.
Innovation Solution
Utilizing a version control platform with branches to ensure atomic updates, where datasets are only provided to downstream processes if all code sets successfully commit, using a first code branch to generate a second branch that produces updated datasets, and updating pointers only if all code sets execute successfully.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If numerous data messages and checks are used to ensure datasets are up-to-date, then reliability is improved, but device complexity and loss of time increase
Solution Approach 1:
A version control platform acts as an intermediary between code sets and downstream processes. The platform manages dataset versions and coordinates updates, eliminating the need for numerous direct data messages and checks between components. This mediator ensures dataset freshness while reducing overall system complexity.
Solution Approach 2:
The system performs preliminary actions by pre-managing dataset versions and preparing update structures before actual data propagation. Version control metadata is prepared in advance, allowing downstream processes to receive datasets without requiring extensive real-time verification messages.
2Reliability
If numerous data messages and checks are used to ensure datasets are up-to-date, then reliability is improved, but loss of time increases
Solution Approach 1:
The version control platform consolidates multiple verification steps into a single coordinated update mechanism. By managing version metadata and update coordination centrally, the platform eliminates redundant time-consuming checks and messages while ensuring datasets are fresh and up-to-date.
3Reliability
If atomic updates are implemented using branches, then reliability is improved, but device complexity increases
Solution Approach 1:
The version control platform creates copies of dataset versions and update structures rather than managing complex in-place updates. By working with versioned copies and metadata references, the system achieves atomic update semantics without requiring complex coordinated modification of shared data structures.
Data Source
AI summary
An apparatus, computer-implemented method and computer program are disclosed for synchronising dataset updates. For example, the method may comprise providing a first code branch associated with a plurality of code sets which, when executed, produce respective time-series datasets for provision to a downstream process linked to the first code branch. The method may also comprise generating a second code branch, based on the first code branch, the second code branch executing the plurality of code sets as part of an updating process and, if successful, storing respective time-series datasets to respective memory locations associated with the second code branch. Another part of the method may comprise determining if all code sets executed by the second code branch have successfully committed. Responsive to a positive determination, one or more pointers, e.g. all pointers, associated with the first code branch may be updated to point to the respective memory locations associated with the second code branch in order that the respective successfully-committed time-series datasets are provided to the downstream process.