Coordinated Node Deduplication with In-Progress Fingerprints

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data storage systems fail to detect identical writes of new data in parallel, leading to multiple copies of the same data being stored without deduplication, resulting in wasted storage space and inefficient resource utilization.

Innovation Solution

Implementing 'in-progress' flags for fingerprint entries in a deduplication index to coordinate deduplication among nodes, allowing nodes to detect and handle identical writes in parallel without significant latency by marking data for backend deduplication checking.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple storage processors perform inline deduplication in parallel, then processing speed is improved, but duplicate data storage occurs due to race conditions in fingerprint database access

Engineering Contradiction:

Improveprocessing speedVSAvoidstorage space waste

Core Design Contradiction:

ProductivityVSLoss of substance

Solution Approach 1:

The system performs preliminary actions by having the first storage processor write data to storage and update the fingerprint database before the second storage processor checks the database. This sequencing prevents the race condition where both processors would independently determine no duplicate exists and write duplicate copies.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses feedback mechanisms where storage processors continuously monitor and update the fingerprint database with newly stored data. When the second storage processor checks the database after the first has updated it, the feedback (presence of the fingerprint entry) informs the second processor that a duplicate already exists, preventing redundant storage.

Inventive Principle:

Principle #23Feedback

2Reliability

If storage processors wait for fingerprint database synchronization, then data integrity is improved, but processing latency increases

Engineering Contradiction:

Improvedata integrityVSAvoidprocessing latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by completing data writing and fingerprint database updates before subsequent processors perform their checks. This eliminates the need for waiting/synchronization while maintaining data integrity, as the database is already in its final state when checked.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses temporary placeholder entries in the fingerprint database during the transition period between data writing and fingerprint registration. These short-living placeholder objects allow processors to quickly check for duplicates without waiting for full synchronization, maintaining both speed and integrity.

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS12430304B2Coordinating deduplication among nodes

Publication Date: 2025.09.30 DELL PROD LP

AI summary

Techniques coordinate deduplication among nodes. The techniques involve, in response to a lookup query to search a deduplication index for an entry that maps a fingerprint which is based on incoming data, detecting an invalid result. The techniques further involve, in response to detecting the invalid result, updating the deduplication index to include an entry that maps the fingerprint, the entry initially storing an “in-progress” flag. The techniques further involve, after the deduplication index is updated to include the entry that maps the fingerprint and after the incoming data is stored in a storage location, removing the “in-progress” flag from the entry.