Segmented Deduplication Index for Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the amount of data blocks stored in non-volatile data resources increases, the size of the deduplication index becomes large, leading to unacceptable memory requirements and performance degradation in data storage systems, as existing technologies require accessing the complete deduplication index for each I/O write request.
Innovation Solution
A segmented deduplication index is implemented, where a resident subset is stored in memory and a non-resident subset is stored in non-volatile data storage, with entries divided into segments based on time periods and usage ratios, allowing for efficient deduplication by comparing digests only within the resident subset and storing new digests in an open segment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the complete deduplication index is stored in memory to enable fast duplicate detection, then deduplication speed is improved, but memory consumption becomes unacceptable
Solution Approach 1:
The deduplication index is divided into multiple segments that are stored in different locations (memory and non-volatile storage). Frequently accessed segments are kept in memory while less frequently accessed segments are stored in non-volatile storage, thus reducing overall memory consumption while maintaining fast access for hot data.
Solution Approach 2:
Different segments of the deduplication index are placed in different storage media based on their access patterns. Memory-resident segments provide fast access for frequently accessed data, while non-volatile storage segments handle less frequently accessed data, optimizing the trade-off between speed and memory usage.
2Quantity of substance
If the complete deduplication index is stored in non-volatile data storage to reduce memory requirements, then memory consumption is reduced, but access speed degrades
Solution Approach 1:
The deduplication index is segmented and distributed across memory and non-volatile storage. By keeping frequently accessed segments in memory, the system maintains fast access speeds for the most important data while storing less frequently accessed segments in non-volatile storage to reduce overall memory requirements.
Solution Approach 2:
The system dynamically manages which index segments reside in memory versus non-volatile storage based on access patterns. Frequently accessed segments are kept in memory while less frequently accessed segments are moved to non-volatile storage, allowing the system to adapt to changing workload requirements and maintain optimal performance.
3Reliability
If all deduplication index segments are accessed for each I/O write request to ensure complete duplicate detection, then deduplication accuracy is improved, but processing time increases
Solution Approach 1:
The deduplication index is divided into segments that can be independently accessed. The system accesses only the relevant segments containing digests of data blocks that could potentially match the incoming data block, rather than scanning the entire index. This segmented access approach maintains complete duplicate detection accuracy while significantly reducing processing time.
Solution Approach 2:
The system pre-organizes the deduplication index into segments based on data block characteristics and access patterns. This preliminary organization allows the system to quickly identify and access only the relevant segments for a given deduplication operation, avoiding the need to scan the entire index and thus reducing processing time while maintaining accuracy.
Data Source
AI summary
A deduplication index is generated having multiple entries, each entry storing a digest of a data block that was previously stored in non-volatile data storage together with a pointer to the location in non-volatile storage at which the data block was previously stored. The entries of the disclosed deduplication index are divided into multiple deduplication index segments. A resident subset of the deduplication index segments is stored in memory of the data storage system. A non-resident subset of the deduplication index segments is stored in non-volatile data storage of the data storage system. Data deduplication is performed for each subsequently received data block for which a digest is generated that matches any one of the digests in the entries of the deduplication index segments that are contained in the resident subset of the deduplication index segments.