Delta Change Index Tables for Storage Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in data replication and migration due to the need to scan full data streams to identify delta changes between backups, leading to increased I/O load and reduced performance.
Innovation Solution
Maintaining delta change information in efficient index tables allows for the identification of only changed data objects and blocks, which are then transmitted to the target storage system, reducing the need for full data stream scanning and improving replication and migration efficiency by using differential encoding and changed object/block tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full data stream scanning is performed to identify delta changes between backups, then complete data replication accuracy is achieved, but I/O load increases and performance decreases
Solution Approach 1:
The system performs preliminary actions by maintaining delta change information in index tables between backup operations. Changed object tables and changed block tables are updated during each backup to record modifications, deletions, and insertions. This preliminary tracking eliminates the need for full data stream scanning in subsequent backups, as the system can directly identify delta changes from the maintained index structures, thereby resolving the contradiction between replication accuracy and performance.
2Reliability
If full data streams are transmitted for each backup, then data completeness is ensured, but network traffic increases unnecessarily
Solution Approach 1:
The system extracts only the necessary delta changes from the full data stream by utilizing maintained index tables. Changed object tables identify which objects have been modified, and changed block tables pinpoint specific modified blocks within those objects. This extraction approach transmits only the essential delta changes rather than complete data streams, ensuring data completeness while minimizing network bandwidth consumption.
3Device complexity
If delta change identification is performed without maintaining change information, then system complexity is reduced, but additional scanning is required impacting performance
Solution Approach 1:
The system implements dynamic index structures that adapt and update with each backup operation. Changed object tables and changed block tables are dynamically maintained, adding new change information and updating existing entries as data modifications occur. This dynamic approach allows the system to efficiently track delta changes without requiring complex predetermined schemas, resolving the contradiction between system simplicity and backup performance.
Data Source
AI summary
According to one embodiment, a first storage system receives a first data stream from a second storage system over a network. The first data stream includes data objects and differential object information identifying at least one data object missing from the first data stream. A difference between the first data stream and a second data stream that has been previously received is determined based on the differential object information, including identifying a data object that has been added, deleted, or modified in view of the second data stream. The first data stream is reconstructed based on the second data stream and the difference between the first data stream and the second data stream, generating a third data stream. The third data stream is stored in a persistent storage device of the first storage system, the third data stream representing a complete first data stream without a missing data object.