Solid State Storage Data Deduplication Hash Table Rebalancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data deduplication using solid state storage devices (SSDs), existing methods face inefficiencies due to uneven bucket filling in hash tables, leading to slowed processing from multiple writes required to determine hash value presence and rebalancing challenges.
Innovation Solution
Generating hash values and comparing them to a volatile memory table, correlating ranges with buckets in non-volatile SSD memory, with rebalancing and additional metadata to optimize storage and retrieval, reducing the need for multiple writes and improving processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple writes are used to determine hash value presence in SSD buckets, then data deduplication reliability is improved, but processing speed deteriorates
Solution Approach 1:
The patent applies preliminary action by using a volatile memory table to store and quickly access hash values before writing to the SSD. The volatile memory table is updated in advance during the deduplication process, allowing fast lookup without requiring multiple write operations to the SSD for the same hash value determination.
Solution Approach 2:
The patent introduces volatile memory as an intermediary between the data processing logic and the SSD storage. This intermediary layer caches hash values and bucket information, enabling fast reads and writes without repeatedly accessing the SSD, thus resolving the contradiction between reliability and processing speed.
2Quantity of substance
If hash tables are filled to high capacity in SSDs, then storage utilization is improved, but bucket distribution becomes uneven and processing slows down
Solution Approach 1:
The patent implements dynamic bucket management where the system monitors bucket fill levels and redistributes hash values when buckets become too full. This dynamic rebalancing maintains even distribution across buckets, preventing performance degradation while maximizing storage utilization.
Solution Approach 2:
The patent changes the parameter of bucket capacity distribution by implementing adaptive bucket management. When buckets approach capacity limits, the system redistributes hash values to balance the distribution, ensuring optimal performance even at high storage utilization levels.
3Productivity
If volatile memory is used for hash table storage, then lookup speed is improved, but memory capacity is limited compared to SSD
Solution Approach 1:
The patent segments the storage system into two parts: volatile memory for fast hash value storage and lookup, and SSD for persistent storage and bulk data retention. This segmentation allows the volatile memory to provide high-speed lookup for frequently accessed data while the SSD provides extensive capacity for the complete hash table and data blocks.
Solution Approach 2:
The patent adds a temporal dimension to the storage architecture by using volatile memory for short-term, high-speed access during active deduplication operations, while relying on SSD for long-term persistence. This dimensional approach in time allows both high speed and large capacity to coexist.
Data Source
AI summary
Systems and methods of data deduplication are disclosed comprising generating a hash value of a data block and comparing the hash value to a table in a first memory that correlates ranges of hash values with buckets of hash values in a second memory different from the first memory. A bucket is identified based on the comparison and the bucket is searched to locate the hash value. If the hash value is not found in the bucket, the hash value is stored in the bucket and the data block is stored in a third memory. The first memory may be volatile memory and the second memory may be non-volatile random access memory, such as an SSD. Rebalancing of buckets and the table, and use of additional metadata to determine where data blocks should be stored, are also disclosed.