Sub-Segment Signature Compression for Faster Data Replication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data replication and backup systems incur high costs and delays due to the need to transfer entire blocks of data over communication networks, even when only a small fraction of the data has changed, as existing methods lack efficient mechanisms to identify and transmit only modified data segments.

Innovation Solution

The system partitions data segments into sub-segments, calculates signatures for each sub-segment using hash functions, and compares these signatures to determine which segments have changed, allowing only modified sub-segments to be transmitted, thereby reducing data transfer over communication networks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If entire blocks of data are transferred over the network for replication and backup, then data integrity is ensured, but bandwidth usage increases and transmission delays occur

Engineering Contradiction:

Improvedata integrityVSAvoidbandwidth usage

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent divides data blocks into smaller sub-segments and calculates individual signatures for each sub-segment. This segmentation allows the system to identify and transfer only the modified sub-segments rather than entire blocks, reducing network bandwidth usage while maintaining data integrity through signature verification.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces signature functions (hash functions) as intermediaries to represent data sub-segments. These signatures serve as compact proxies that enable comparison and change detection without transferring the actual data, thus reducing bandwidth consumption while ensuring data integrity through signature matching.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If entire blocks of data are transferred over the network, then complete data replication is achieved, but transmission time increases

Engineering Contradiction:

Improvedata completenessVSAvoidtransmission delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

By segmenting data blocks into sub-segments and using individual signatures for each, the system can identify and transfer only the changed portions. This significantly reduces transmission time compared to sending entire blocks, while maintaining data completeness through signature-based change detection.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts only the modified sub-segments from complete data blocks for transmission. By using signature comparison to identify changes, the system removes unnecessary data from the transfer process, reducing transmission time while ensuring complete replication of only the necessary data portions.

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If signature functions are used to identify changed sub-segments, then data transfer efficiency improves, but computational overhead increases

Engineering Contradiction:

Improvedata transfer efficiencyVSAvoidcomputational overhead

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent transforms large data sub-segments into compact signature representations (changing the parameter from bytes to bits). This transformation enables efficient comparison and change detection with minimal computational overhead, as signature calculation and comparison are much less resource-intensive than processing the actual data bytes.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent creates signature copies of data sub-segments that can be quickly compared without manipulating the original large data. These signature copies serve as lightweight proxies for change detection, improving data transfer efficiency while minimizing computational overhead compared to direct data comparison.

Inventive Principle:

Principle #26Copying

4Loss of energy

If only modified sub-segments are transmitted, then bandwidth usage is reduced, but system complexity increases

Engineering Contradiction:

Improvebandwidth usageVSAvoidsystem complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The patent introduces signature functions as intermediaries that simplify the complexity of identifying changed data. Instead of complex direct comparison of large data sub-segments, the system uses simple signature calculation and comparison, reducing system complexity while enabling selective transmission of only modified portions to save bandwidth.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS8205009B2Apparatus for continuous compression of large volumes of data

Publication Date: 2012.06.19 EMC IP HLDG CO LLC

AI summary

A system for efficiently transmitting data from a first site to a remote site over a communication medium. The data includes a storage for storing data in sub-segment boundaries, such that few sub-segments are accommodated in each block. The system further includes a storage for storing data including signature data. Each one of the sub-segments is associated with a signature of considerably smaller size than its respective sub-segment. The system includes a processor configured to perform the following, as many times as required: receiving a block and partitioning it into sub-segments. For each sub-segment in the block the processor calculating a signature. It then determines whether the calculated signature matches a corresponding signature, if any, stored in the signature storage, and in case of no match (indicating that the sub-segment is new or has been modified), transmitting the sub-segment to the remote site and store the calculated signature in the signature storage.