Data Consistency via Reference High Water Mark
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
During the archiving process from a transactional data store to a non-transactional archive data store, inconsistencies such as duplicate, missing, or incomplete data can occur due to intermittent failures, leading to inconsistent views when queries are made during the transitional period.
Innovation Solution
The implementation of a reference high water mark (HWM) is used to manage data availability by preventing queries from viewing new data until the archiving operation completes, ensuring that only complete and correct data is visible in both the transactional and archive data stores, achieved through query rewriting and data hiding techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of stationary object
If data is copied from transactional data store to archive data store during transitional period, then long-term data archiving is enabled, but data consistency and completeness deteriorate due to intermittent failures and duplicate/missing data
Solution Approach 1:
The patent applies preliminary action by copying data to a staging area before the actual archive data store, and by pre-computing and storing dependency information about which records depend on which. This allows the system to prepare data for archiving in advance and to identify potential consistency issues before they affect query results, thereby maintaining data consistency during the archiving process.
Solution Approach 2:
The patent introduces a staging area as an intermediary between the transactional data store and the archive data store. This intermediate layer receives copied data, validates its completeness, and only promotes it to the archive store when consistency is confirmed. The intermediary also manages the dependency resolution process, ensuring that archived data remains consistent with the transactional store during transitions.
2Productivity
If data is copied and deleted during transitional period, then data archiving progress is improved, but query accuracy deteriorates due to duplicate, missing, or incomplete data views
Solution Approach 1:
The system pre-computes dependency relationships between records before archiving begins and stores this information in the staging area. When queries are executed during the archiving transition, the system uses these pre-computed dependencies to determine which records should be visible and which should be excluded, ensuring query accuracy matches the intended data state without waiting for archiving to complete.
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously monitors the archiving process and adjusts query results accordingly. When data is copied to the staging area, the system tracks this state and uses the information to filter query results, ensuring that queries always return consistent data views regardless of where the archiving process stands in its progression.
3Ease of operation
If queries are allowed during archiving operation, then system responsiveness is maintained, but data view consistency deteriorates due to intermittent failures and transitional state
Solution Approach 1:
The staging area serves as an intermediary that receives data copies and validates them before making them available to queries. This intermediary layer ensures that only complete and consistent data is made visible to queries, even during active archiving operations. The system can thus maintain responsiveness by allowing queries to execute against the staging area data while preserving consistency through the intermediary's validation process.
Solution Approach 2:
The system performs preliminary validation and dependency resolution in the staging area before data is made queryable. By pre-processing and verifying data completeness and relationship integrity in advance, the system ensures that queries executed during archiving operations receive consistent results, eliminating the need to choose between responsiveness and consistency.
Data Source
AI summary
Embodiments regard maintaining data consistency between transactional and non-transactional data stores. Embodiments may include receiving a request to archive a set of data stored in a first data store, recording a high water mark in the first data store for the set of data, and archiving the set of data. Archiving the set of data may include copying the set of data into a second data store and inserting a high water mark for the set of data in the first data store and the second data store. Embodiments may further include updating a reference high water mark upon successful completion of operations for the archiving of the data, the reference high water mark representing completion of a last successful archive operation, and, upon receiving a query for requested data contained in the first or second data store, determining whether to allow viewing of the requested data based at least in part on a current value of the reference high water mark.