Hash Table Virtual Buckets for Memory Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication methods in memory systems, particularly in volatile memory like DRAM, face challenges in efficiently reducing redundant data copies, leading to increased read latency, memory bandwidth usage, and reduced write endurance, while also lacking fine-grained deduplication capabilities.
Innovation Solution
A memory deduplication method using a hash table system with physical and virtual buckets, where data is hashed and stored in a three-dimensional array, allowing for efficient movement of blocks between adjacent virtual buckets to free up space and reduce duplicates, and utilizing a processor to manage deduplication without external commands, thereby increasing memory capacity and reducing redundant data copies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional in-memory deduplication technology is used with CPU-centric approach, then data deduplication can be achieved, but read latency increases and memory bandwidth is consumed
Solution Approach 1:
The hash table is segmented into multiple levels (primary hash table, secondary hash tables, tertiary hash tables) with each level handling different aspects of deduplication. This segmentation allows parallel processing of hash computations and data comparisons, reducing read latency while effectively identifying and eliminating redundant data copies.
Solution Approach 2:
The patent introduces a multi-dimensional hash table structure with primary, secondary, and tertiary levels, adding spatial dimensions to the traditional single-level hash table. This dimensional expansion enables simultaneous access to multiple hash buckets through different paths, reducing the time required to identify duplicates and decreasing read latency.
2Measurement precision
If data deduplication is implemented in volatile memory, then fine-grained deduplication capability is achieved, but device complexity increases
Solution Approach 1:
The complex deduplication task is segmented across multiple hash table levels, with each level responsible for specific granularities of data comparison. The primary hash table handles coarse-grained deduplication while secondary and tertiary tables handle finer granularities, making the overall complex system manageable through modular organization.
Solution Approach 2:
The hash table structure is designed to be dynamic and adaptive, with entries that can be efficiently inserted, updated, and removed based on data access patterns. The multi-level structure allows the system to dynamically adjust its deduplication granularity and memory usage, managing complexity through flexibility rather than rigid structure.
3Productivity
If multiple hash tables with physical and virtual buckets are used, then deduplication efficiency is improved, but memory bandwidth usage increases
Solution Approach 1:
Multiple hash tables (primary, secondary, tertiary) are merged into a unified deduplication system that shares memory resources and processing logic. The virtual bucket concept allows multiple physical buckets to be logically combined, reducing redundant memory accesses and optimizing bandwidth usage while maintaining high deduplication efficiency through coordinated operation of all tables.
Data Source
AI summary
A method of memory deduplication includes identifying hash tables each corresponding to a hash function, and each including physical buckets, each physical bucket including ways and being configured to store data, identifying virtual buckets each including some physical buckets, and each sharing a physical bucket with another virtual bucket, identifying each of the physical buckets having data stored thereon as being assigned to a single virtual bucket, hashing a data line according to a hash function to produce a hash value, determining whether a corresponding virtual bucket has available space for a block of data according to the hash value, sequentially moving data from the corresponding virtual bucket to an adjacent virtual bucket when the corresponding virtual bucket does not have available space until the corresponding virtual bucket has space for the block of data, and storing the block of data in the corresponding virtual bucket.