Dynamic Deduplication Granularity for Memory Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current memory systems face challenges such as poor write latency, decreased bandwidth, high cost-to-bit ratio, complicated non-volatile memory management, and ineffective data compaction due to metadata overhead from techniques like DRAM deduplication, which fail to reliably increase memory capacity and deduplication ratios.
Innovation Solution
The proposed algorithm architecture employs three main techniques: dynamic deduplication granularity selection, efficient hash table and translation table management, and variable reference counter sizes to reduce metadata overhead and improve deduplication ratios without burdening memory system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If DRAM deduplication with hash tables, reference counters, and translation tables is used to increase virtual memory capacity, then memory capacity is increased, but metadata overhead increases and undermines the original intent of deduplication
Solution Approach 1:
The patent extracts the reference counter functionality from the traditional metadata structure and implements it as a separate bitmap-based mechanism. This separation removes the overhead burden from the hash table and translation table structures, allowing them to focus solely on storing unique data content mappings while the bitmap handles reference tracking independently and more efficiently.
Solution Approach 2:
The patent changes the parameter representation of reference counters from traditional integer-based counters to bitmap-based flags. This parameter transformation reduces the space required per reference counter from multiple bytes to a single bit per reference, dramatically reducing metadata overhead while maintaining the ability to track and manage referenced data lines.
2Reliability
If traditional reference counters are used to track unique data lines, then data uniqueness is tracked, but memory bandwidth and write latency deteriorate
Solution Approach 1:
The patent replaces the mechanical increment/decrement operation of traditional reference counters with a bitwise set/clear operation on bitmap flags. This substitution eliminates the need for complex counter arithmetic and memory access patterns, replacing them with simple, fast bitwise operations that can be executed more efficiently and with less impact on memory bandwidth.
3Quantity of substance
If fine-grained deduplication is applied to increase deduplication ratio, then more data can be compacted, but metadata overhead and processing complexity increase
Solution Approach 1:
The patent segments the memory space into distinct regions, each with its own deduplication granularity and bitmap. This segmentation allows different parts of memory to use different deduplication strategies optimized for their specific workloads, enabling fine-grained control over deduplication ratios without requiring a single complex global metadata structure to manage all granularities.
Data Source
AI summary
A method of dynamically selecting deduplication granularity in a memory system to decrease deduplication granularity and to increase hash-table efficiency, the method including selecting one or more deduplication granularities at an application level of an application using the memory system, the one or more deduplication granularities being selected according to features of the memory system, and assigning a memory region corresponding to each of the one or more selected deduplication granularities, where the method may use a memory manager to share memory translation table and hash table, and may be employed by a system that enables using higher capacity pre-allocated counter fields for frequently utilized lines.