Data Storage Change Reporting for Near-Zero RPO Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in achieving Near-Zero Recovery Point Objective (RPO) due to performance limitations in identifying and replicating changed data, particularly when using snapshot differencing techniques, which require expensive flushes of cached data.
Innovation Solution
A data storage system employs a tracking structure with a b-tree mechanism to track and merge write transactions, creating reporting entries that efficiently identify and replicate changed data extents, reducing the number of elements required to describe changed data over an RPO cycle.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If snapshot differencing techniques are used to identify changed data, then data replication can be performed, but expensive flushes of cached data are required which limits achieving Near-Zero RPO
Solution Approach 1:
The system segments the tracking of data changes by using a tracking structure with tracking entries for individual write transactions, separating the tracking phase from the reporting phase. This allows continuous tracking of writes without requiring expensive cache flushes, as only the tracking entries need to be processed periodically to generate replication reports.
Solution Approach 2:
The system performs preliminary tracking of write transactions as they occur, storing them in tracking entries within the tracking structure. This preliminary action allows the system to prepare change information in advance without requiring immediate cache flushes or expensive operations, enabling Near-Zero RPO while maintaining replication performance.
2Measurement precision
If individual write transactions are tracked separately, then accurate change tracking is achieved, but the number of elements required to describe changed data increases
Solution Approach 1:
The system merges multiple tracking entries into reporting entries by processing tracking entries to identify and combine changes to the same data extent. The reporting structure consolidates individual write tracking information into summarized reporting entries that describe changed data extents, reducing the number of elements while maintaining accurate change tracking through the merging characteristic.
Data Source
AI summary
Selective replication of data in a data storage system includes tracking write transactions using tracking entries in a tracking structure. The tracking structure is periodically processed to create a reporting structure containing reporting entries each identifying a respective extent modified by one or more of the write transactions. The processing for each tracking entry includes (1) examining the reporting structure to determine whether the tracking entry expands an extent of a reporting entry, (2) based on the tracking entry not expanding an extent a reporting entry, creating a new reporting entry based on the tracking entry, and (3) based on the tracking entry expanding an extent of a reporting entry, modifying the reporting entry to identify the extent as so expanded. The reporting structure is used to perform a replication copy operation of the extents identified therein.