Continuous Data Replication with Deduplication and Fingerprint Verification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data protection systems face challenges such as system shutdown during backups, limited recovery points, and lengthy data recovery processes, especially when using periodic backups or data replication methods.

Innovation Solution

A system and method for continuous data replication on deduplicated storage, which includes creating an initial backup snapshot, maintaining journals for point-in-time recovery, and synthesizing new backup snapshots using IOs from multiple data protection appliances, allowing for continuous data protection and reduced storage consumption.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If periodic backup is used, then storage cost is reduced, but data recovery time increases and recovery points are limited

Engineering Contradiction:

Improvestorage capacityVSAvoiddata recovery time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The backup system segments data into blocks and maintains a catalog of data blocks with metadata. Instead of periodically backing up entire datasets, the system segments backup operations into individual block-level operations, allowing selective and efficient data recovery without requiring full dataset restoration.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system creates fingerprints (hash values) as copies of data block identifiers and stores them in a catalog. These fingerprint copies enable rapid data verification and recovery by comparing stored fingerprints with actual data blocks, eliminating the need to restore entire datasets for verification purposes.

Inventive Principle:

Principle #26Copying

2Reliability

If data replication is used, then data availability is improved, but storage consumption increases

Engineering Contradiction:

Improvedata availabilityVSAvoidstorage capacity

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The system merges duplicate data blocks from multiple sources into a single stored copy, tracking references to that copy in a catalog. When multiple data blocks are identical, the system stores one copy and maintains metadata references, thereby reducing total storage consumption while maintaining data availability through the catalog references.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The system changes the storage parameter from storing complete data copies to storing data block fingerprints (hash values) in the catalog. This parameter change from full data replication to fingerprint-based tracking dramatically reduces storage consumption while maintaining the ability to verify and recover data through fingerprint comparison.

Inventive Principle:

Principle #35Parameter changes

3Stability of the object's composition

If system shutdown is performed during backup, then data consistency is ensured, but productivity decreases

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem availability

Core Design Contradiction:

Stability of the object's compositionVSProductivity

Solution Approach 1:

The system performs preliminary actions by creating and storing fingerprints of data blocks before actual data transfer or modification. The catalog is updated with data block metadata and fingerprints in advance, allowing the system to verify data consistency through fingerprint comparison without requiring system shutdown during the backup operation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system replaces the mechanical approach of system shutdown for consistency checking with a software-based fingerprint comparison mechanism. By substituting physical system state changes (shutdown) with computational verification (fingerprint hashing and comparison), the system maintains data consistency while keeping the production system operational.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10484179B1Data consistency in an encrypted replication environment

Publication Date: 2019.11.19 DELL EMC

AI summary

A System, Computer Program Product, and computer-executable method for testing data consistency in a replicated data storage environment, wherein the replicated data storage environment includes a production site and a target site, wherein the target site is located within a cloud storage provider, the System, Computer Program Product, and computer-executable method includes receiving a request for a data consistency check of encrypted data stored at the target site, retrieving replicated signatures from the target site, retrieving production signatures from the production site, and determining data consistency based on an analysis of the replicated signatures and the production signatures.