Multi Source Wire Deduplication via Digest Copying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data protection systems face challenges in maintaining continuous data protection, particularly in keeping pace with high data transaction rates at production sites without slowing them down, leading to potential shutdowns due to backlog issues at backup sites.
Innovation Solution
A method for data deduplication in a replication environment that uses a global memory address space over a WAN, enabling an eviction mechanism to minimize data transmission and maintain data integrity even when write ordering between sources is not guaranteed, allowing for efficient data replication from multiple sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If journaling is used to enable continuous data protection and rollback capability, then the ability to recover to any specified point in time is improved, but the overhead of multiple data transactions at the backup site causes the backup site to fall behind when data transaction rates are high
Solution Approach 1:
The patent uses digest copying instead of full data copying. The production site calculates digests of data chunks and sends only these digests to the backup site for logging, rather than sending complete data chunks. This dramatically reduces the transaction overhead at the backup site while maintaining the ability to recover to any point in time.
Solution Approach 2:
The patent extracts only the essential identification information (digests) from the complete data chunks. By taking out just the digest portion and leaving the full data at the production site, the system reduces backup site processing requirements while preserving rollback capability through journaling.
2Reliability
If the production site is forced to slow down to allow the backup site to keep pace with data transactions, then the backup site can process transactions without backlog, but the production site availability is reduced
Solution Approach 1:
By copying only digests rather than full data chunks to the backup site, the patent enables the production site to maintain high transaction rates without overwhelming the backup site. The reduced data volume allows the backup site to process transactions at normal speeds without requiring production site throttling.
3Reliability
If data is transmitted frequently to the backup site to maintain continuous protection, then recovery point objective is improved, but data transmission overhead increases
Solution Approach 1:
The patent transmits only digest copies rather than complete data chunks to the backup site. This copying approach maintains continuous data protection and improves recovery point objective while dramatically reducing transmission overhead and energy consumption.
Solution Approach 2:
The patent applies partial action by transmitting only the necessary portion (digests) of the data rather than complete chunks. This partial transmission is sufficient to maintain continuous protection capability while avoiding the excessive overhead of full data replication.
Data Source
AI summary
A method, system, and program product for data deduplication in a replication environment, the replication environment having a production site, a splitter, and a replication site, wherein the replication site has a journal, comprising determining a digest for each chunk of data of a set of data chunks, determining for each chunk whether the digest is in an index on the production site, determining for which offsets are to be evicted from the cache on the replication site, replacing the chunks in set of chunks that are in the index with an offset, and transmitting the set of chunks, offsets and an eviction list to the replication site.