Consistency Point Markers for Data Copy Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in determining when a target system has a consistent copy of data, as it is difficult to establish and maintain a consistent state without negatively impacting the source system or applications accessing the data.
Innovation Solution
The implementation of consistency point markers and a framework that captures and processes file operations in real-time, using components like the Volume Shadow Copy Service (VSS) to pause transactions, flush cached data, and insert markers into a stream of file operations, allowing for the determination of when the target system has a consistent copy of the source data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-time capture and processing of file operations is implemented to determine consistent data states, then data consistency is improved, but system complexity increases
Solution Approach 1:
A consistency determination component is introduced as an intermediary between the file operation stream and the target data system. This component captures file operations, processes them through a consistency determination framework, and generates consistency markers without requiring direct intervention in the source or target systems, thereby maintaining data consistency while managing system complexity.
Solution Approach 2:
The system creates a virtual copy of the file operation stream and processes this copy through the consistency determination framework. By working with a replicated stream rather than the original live operations, the system can analyze and determine consistency states without impacting the actual data flow or source system performance.
2Measurement precision
If consistency markers are inserted into the file operation stream to track consistent states, then consistency determination is improved, but processing overhead increases
Solution Approach 1:
The consistency determination framework performs preliminary processing of file operations as they are captured, analyzing and evaluating consistency conditions before the operations are applied to the target data. This advance preparation allows consistency markers to be generated proactively, reducing the need for time-consuming post-processing and validation operations.
Solution Approach 2:
The system implements optimized processing paths that skip unnecessary consistency checks for operations that cannot affect data consistency, such as read operations or operations on already-processed data. By selectively applying consistency determination only where necessary, the system maintains measurement precision while minimizing processing overhead and time loss.
Data Source
AI summary
Described are a method, computer program product, and system for obtaining a copy of source data in a consistent state. One or more file operations having a corresponding time sequence which modify said source data are recorded. A request for a copy of the source data in a consistent state is received. It is determined at which point in the corresponding time sequence said source data is in a consistent state as a result of applying a portion of the file operations. The point in the corresponding time sequence at which the source data is in a consistent state is marked. The portion of file operations determined to place the source data in a consistent state is applied to the copy of the source data.