Data Deduplication Using XOR Similarity Comparison

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current storage systems face inefficiencies due to the high cost and limited availability of solid-state storage devices, necessitating the use of methods like deduplication and compression to maximize storage efficiency.

Innovation Solution

A computer-implemented method that identifies potential deduplication candidates and targets, executes a comparison operation using XOR operations to determine similarity, and stores comparison results for subsequent reconstruction, enabling efficient storage and retrieval of data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If deduplication and compression methods are employed to maximize storage efficiency, then storage cost is reduced, but system complexity increases

Engineering Contradiction:

Improvestorage capacityVSAvoidsystem complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The system performs preliminary hashing of data blocks before storage to create unique identifiers. This advance preparation enables efficient deduplication by allowing the system to quickly identify and eliminate duplicate blocks without complex real-time comparisons, thereby reducing storage requirements while maintaining manageable system complexity

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces hash functions as intermediary elements that transform original data blocks into compact representations. These hash values serve as mediators that enable efficient comparison and identification of duplicate data without requiring direct comparison of the actual data blocks, thus achieving storage efficiency while keeping the system architecture relatively simple

Inventive Principle:

Principle #24Intermediary (Mediator)

2Quantity of substance

If complex methodologies are utilized to navigate technology limitations, then storage efficiency is improved, but processing time increases

Engineering Contradiction:

Improvestorage efficiencyVSAvoidprocessing time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent divides data into discrete blocks and applies hashing to each block independently. This segmentation allows parallel processing of multiple data blocks simultaneously, reducing overall processing time while maintaining high storage efficiency through systematic deduplication of individual blocks

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system replaces complex mechanical comparison operations with mathematical hash function computations. This substitution enables faster processing by using computationally efficient hash calculations instead of time-consuming byte-by-byte comparisons, thereby improving storage efficiency without excessive processing time penalties

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10963437B2System and method for data deduplication

Publication Date: 2021.03.30 EMC IP HLDG CO LLC

AI summary

A method, computer program product, and computing system for identifying a potential deduplication candidate and a related deduplication target; executing a comparison operation with respect to the potential deduplication candidate and the related deduplication target to generate a comparison result; and determining a level of similarity between the potential deduplication candidate and the related deduplication target by processing the comparison result.