Parallel Data Replication with Deduplication and Checksum Verification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data replication methods are inefficient as they transfer and verify data in a sequential manner, leading to delayed error detection and increased storage requirements due to lack of deduplication.

Innovation Solution

Implementing a system that parallel-transmits and verifies data portions, combines verified data, and deduplicates data before storage, using a deduplication service with fingerprinting and checksums to identify unique segments and reduce storage needs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data is transferred and verified in sequential manner, then verification accuracy is maintained, but data transfer efficiency deteriorates

Engineering Contradiction:

Improveverification accuracyVSAvoiddata transfer efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent divides data into multiple portions and processes them in parallel. Each portion is independently verified using checksums and fingerprints, allowing simultaneous verification of multiple data segments without compromising accuracy. This segmentation enables parallel processing while maintaining verification reliability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent calculates fingerprints and checksums for data portions before transmission. By preparing verification data in advance, the system can immediately verify received portions without waiting for complete data transfer, enabling early error detection and parallel processing of subsequent portions.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If complete data is transferred before verification, then verification thoroughness is ensured, but error detection timing deteriorates

Engineering Contradiction:

Improveverification thoroughnessVSAvoiderror detection timing

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent calculates verification data (fingerprints and checksums) for each data portion before transmission. This preliminary preparation allows the receiving system to verify each portion immediately upon receipt, enabling error detection during the transfer process rather than waiting for complete data arrival.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

By dividing data into multiple verifiable portions with individual checksums and fingerprints, the system can perform thorough verification on each segment independently. This segmentation allows comprehensive verification without requiring waiting for complete data transfer, thus maintaining thoroughness while improving detection timing.

Inventive Principle:

Principle #1Segmentation

3Ease of manufacture

If data is stored without deduplication, then storage simplicity is maintained, but storage efficiency deteriorates

Engineering Contradiction:

Improvestorage simplicityVSAvoidstorage requirements

Core Design Contradiction:

Ease of manufactureVSQuantity of substance

Solution Approach 1:

The patent uses fingerprinting (hash functions) to create unique identifiers for data portions. These fingerprints act as digital signatures that enable the system to identify duplicate data segments. When a fingerprint matches an existing entry, the system recognizes the data as duplicate and avoids redundant storage, significantly reducing storage requirements while maintaining simple storage operations.

Inventive Principle:

Principle #32Color changes

Data Source

PatentUS10929382B1Method and system to verify integrity of a portion of replicated data

Publication Date: 2021.02.23 EMC IP HLDG CO LLC

AI summary

In general, embodiments of the invention relate to methods and systems for replicating data, which is stored in a source system, in a target system. More specifically, embodiments of the invention enable parallel transmission and verification of portions of the data. Once the portions of the data have been verified, embodiments of the invention combine the verified portions of the data to obtain the final combined data. The combined data is then verified.