Distributed Storage Deduplication via Hash Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods in distributed computer storage systems face challenges in efficiently managing duplicate data elimination during write operations across multiple computer devices, leading to storage space inefficiencies and complex metadata management.
Innovation Solution
The proposed solution involves a method where each computer device computes a hash value for incoming data, divides it into parts, and assigns a unique deduplication ID, using a deduplication database to determine existing entries. If duplicates are found, it creates virtual address entries and pointers to storage locations, managing reference counters and validity indicators to optimize data storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is implemented in a distributed storage system, then storage space efficiency is improved, but metadata management complexity increases
Solution Approach 1:
The patent segments the deduplication process into distinct components: hash computation units that generate fingerprints, a deduplication database that stores metadata, and virtual address management that handles mapping. This segmentation allows each component to be optimized independently, reducing overall metadata management complexity while maintaining storage efficiency.
Solution Approach 2:
The patent introduces virtual address entries as intermediaries between logical block addresses and physical storage locations. These virtual address entries serve as a mediation layer that simplifies metadata management by providing a uniform interface for deduplication operations across the distributed system, reducing the complexity of direct metadata handling.
2Reliability
If deduplication IDs are assigned to be unique within logical block address space, then data uniqueness is ensured, but storage overhead increases
Solution Approach 1:
The patent moves the deduplication ID assignment to a new dimensional space by using hash value parts (H1 and H2) derived from cryptographic hashing of the data. Instead of assigning sequential IDs within the logical block address space, the system uses the hash-based dimension which provides unique identification with minimal overhead, as the hash values are computed from the data itself rather than requiring additional sequential numbering.
3Speed
If virtual address entries with pointers are created for duplicate data, then data retrieval is optimized, but memory usage increases
Solution Approach 1:
The patent merges multiple virtual address entries that point to the same physical storage location by using the deduplication database to track and consolidate references. When duplicate data is detected, the system creates virtual address entries that point to the existing stored data rather than creating new storage locations, thereby optimizing retrieval while minimizing additional memory consumption through reference consolidation.
Data Source
AI summary
The presently disclosed subject matter includes various inventive aspects, which are directed for enabling execution of deduplication during data writes in a distributed storage-system.