Dedupe Storage Network Replication via Chunk Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection systems face challenges in optimizing replication of backup images across multiple sites, particularly in deduplication storage networks, where efficient replication and storage solutions are needed to manage data retention, disaster recovery, and storage space optimization.
Innovation Solution
A dedupe storage network with onsite and replication-site dedupe storage nodes, utilizing a mesh-network topology and inline deduplication file systems that convert incoming backup streams to dedupe images without staging, enabling chunk-wise deduplication, compression, and encryption, while maintaining metadata and data elements in local and remote layouts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If backup images are replicated across multiple sites, then data retention and disaster recovery capability are improved, but storage space consumption increases
Solution Approach 1:
The backup image is segmented into multiple chunks, and only unique chunks are replicated across sites. The system divides the backup image into chunk segments, calculates fingerprints for each chunk, and identifies duplicate chunks that can be eliminated, thereby reducing storage space while maintaining data integrity for disaster recovery
Solution Approach 2:
The system changes the representation of backup data by using deduplication parameters and chunk-wise processing. By transforming the backup image into a deduplicated format with metadata elements and data elements, the system reduces storage requirements while preserving the ability to reconstruct complete images for disaster recovery
2Quantity of substance
If deduplication is implemented in backup streams, then storage efficiency is improved, but processing complexity increases
Solution Approach 1:
The deduplication process is performed preliminarily during the backup stream processing stage. The inline deduplication file system processes backup streams in real-time, identifying and eliminating duplicate chunks before data is written to storage, thereby simplifying subsequent storage operations while achieving high storage efficiency
Solution Approach 2:
The system uses fingerprint-based copying to identify duplicate chunks. By calculating fingerprints of backup stream chunks and comparing them against existing chunks, the system efficiently identifies duplicates without complex processing, achieving storage efficiency through simple comparison operations
3Loss of information
If remote replication is implemented, then data availability is improved, but network bandwidth consumption increases
Solution Approach 1:
The system extracts only the unique chunks from backup images for replication to remote sites. By identifying and extracting only the non-duplicate portions of backup data, the system reduces network bandwidth consumption while ensuring that complete data sets are available at remote locations for data availability and disaster recovery
Data Source
AI summary
In one embodiment, a computer-implemented method of a dedupe storage network includes the step of providing one or more replication-site dedupe storage nodes. Another step includes providing an onsite-dedupe storage node, wherein the onsite dedupe node initiates a replication operation, wherein the replication operation comprises the onsite dedupe storage node operation pushing or fetching a dedupe-image data from the one or more replication-site dedupe storage nodes. The replication from local dedupe storage node to remote dedupe storage node pushes data not present at remote. The replication from remote dedupe storage node to local dedupe storage node fetches data not present at local.