Hash-Based Data Placement for Distributed Storage Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Shared nothing systems face inefficiencies due to duplicate data placement across nodes, which reduces data reduction efficiency across a cluster of machines.

Innovation Solution

The implementation of a hash-based data placement strategy, where a hash of newly written data is generated to place data blocks on nodes already containing similar data, combined with data deduplication techniques to eliminate redundant data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If data is placed on nodes using traditional shared nothing systems, then data distribution is simplified, but duplicate data placement occurs reducing storage efficiency

Engineering Contradiction:

Improvedata reduction efficiencyVSAvoiddata placement complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system performs preliminary actions by computing hashes of data blocks before placement and proactively placing data on nodes that already contain similar data. The placement decision is made in advance based on hash comparisons, ensuring duplicate reduction before data is actually stored.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses feedback mechanisms where nodes provide information about their existing data content (through hash comparisons) to the placement decision process. This feedback loop allows the system to adjust placement decisions based on actual node content, reducing duplicates while maintaining efficiency.

Inventive Principle:

Principle #23Feedback

2Quantity of substance

If data deduplication is implemented, then storage resource utilization is improved, but system complexity increases

Engineering Contradiction:

Improvestorage resource utilizationVSAvoidsystem complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The system extracts and removes duplicate data from the storage process by identifying redundant data blocks through hash comparisons and excluding them from placement. Only unique data blocks are stored, while duplicates are eliminated through the placement strategy itself.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system changes the parameter of data placement from random or simple distribution to hash-based similarity-driven placement. By transforming the placement criterion to consider data content similarity through hash values, the system achieves deduplication without requiring complex post-processing.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If hash-based placement is used to reduce duplicates, then data reduction efficiency improves, but computation requirements increase

Engineering Contradiction:

Improvedata reduction efficiencyVSAvoidcomputation requirements

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system applies partial action by computing hashes only for data blocks that need to be placed, rather than hashing all data continuously. It performs hash comparisons only between candidate data blocks and existing node content, limiting computation to necessary portions and reducing overall computational overhead.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250156075A1Similarity based compression in a distributed storage system

Publication Date: 2025.05.15 PURE STORAGE INC

AI summary

A subset of nodes including storage devices having similar data to data to be stored in a storage system are identified by a storage system controller using hashes corresponding to the data to be stored. The data to be stored in the storage system is transmitted to a particular node of the subset of nodes having data similar to the data based on node characteristics received from the particular node.