Deduplication in Eventually Consistent Storage via Hash Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In eventually consistent distributed data storage systems, data deduplication methods face challenges in safely deleting deduplicated data due to race conditions and the need to avoid sending existing data over the network, as clients may not coordinate their operations, leading to inconsistencies and inefficiencies.
Innovation Solution
The system calculates a hash for incoming data, writes it to an object for deduplication, tracks reference counts, and delays deletion to ensure safety, using idempotent operations and hierarchical naming to manage references and avoid race conditions, allowing for efficient data management and storage optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented in an eventually consistent distributed storage system, then storage space efficiency is improved, but race conditions occur during deletion operations causing data inconsistency
Solution Approach 1:
The system performs preliminary actions by checking hash values against a dictionary before writing data, and by maintaining reference counts before deletion operations. This allows the system to identify duplicate data in advance and safely determine when deduplicated data can be deleted, preventing race conditions while maintaining storage efficiency.
Solution Approach 2:
The patent introduces intermediary mechanisms including a dictionary that stores hash values as intermediaries between data writing operations, and reference counts that act as intermediaries between multiple clients and the deduplicated data. These intermediaries enable safe coordination without requiring direct client coordination, resolving the race condition problem while preserving deduplication benefits.
2Adaptability or versatility
If clients write data without coordination in an eventually consistent system, then system availability and partition tolerance are improved, but duplicate data may be stored across replicas increasing storage requirements
Solution Approach 1:
The system uses cryptographic hash values as copies of data identifiers stored in a dictionary. Instead of storing actual data copies across all replicas, the system stores compact hash representations that enable duplicate detection. This allows clients to write data independently while the system efficiently identifies and eliminates actual data duplicates, maintaining both availability and storage efficiency.
3Quantity of substance
If deletion of deduplicated data is performed immediately when reference count reaches zero, then storage optimization is improved, but race conditions may cause deletion of data still being accessed
Solution Approach 1:
The system performs preliminary checks by maintaining reference counts that track all clients referencing deduplicated data before deletion. The deletion operation is delayed until the reference count reaches zero, ensuring that no client is actively accessing the data. This preliminary tracking mechanism enables safe storage optimization without risking premature deletion.
Solution Approach 2:
The patent implements feedback mechanisms through reference counts that continuously monitor data usage. When a client writes or reads deduplicated data, the reference count is updated, providing feedback about current data usage. This feedback loop enables the system to make informed deletion decisions, optimizing storage while ensuring safety by only deleting when the reference count indicates no active usage.
Data Source
AI summary
Aspects of the present invention include a method, system and computer program product for performing data deduplication for eventually consistent distributed data storage (DDS) system. The method includes receiving data content from one or more clients by a DDS system, wherein the one or more clients do not coordinate transmitting of the data content. The method also includes calculating a hash for the data content by the distributed data storage system, writing the data content to an object used for data deduplication, wherein a name of the object is based on the hash and determining whether the data content is present in the distributed data storage system based on the name of an object previously stored on the DDS system. The method further includes keeping track of a number of references to the data content and delaying deletion of the data content for a predetermined period of time.