Deduplication Digest Cache Prefetching via Page Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in data deduplication processing, particularly in accessing and managing large deduplication data stores due to costly access times and the need for persistent caching solutions that can efficiently handle the large size of deduplication data stores.
Innovation Solution
The implementation of a deduplication data store organized into pages based on spatial and temporal locality criteria, with a deduplication digest cache and mapping structure stored in memory, allowing for efficient prefetching and logical contiguity of entries in the deduplication data store, and a current page mechanism for adding new entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large deduplication data store is maintained in persistent storage, then data durability and capacity are improved, but access time and processing efficiency deteriorate
Solution Approach 1:
The deduplication data store is segmented into pages, with a separate cache structure holding frequently accessed pages in memory. This segmentation allows the system to maintain the complete data store in persistent storage for durability while only keeping active pages in fast memory for quick access, resolving the contradiction between full data availability and access speed.
Solution Approach 2:
A cache structure acting as an intermediary layer is introduced between the persistent storage and the processing logic. This cache holds digests and page mappings in memory, providing fast access to frequently used data while the complete data store remains in persistent storage, thus mediating between the needs for durability and speed.
2Speed
If the entire deduplication data store is cached in memory, then access speed is improved, but memory usage and cost increase
Solution Approach 1:
The data store is divided into manageable pages, and only the necessary pages are loaded into the cache structure in memory. This segmentation enables the system to achieve fast access speeds for active data while keeping memory usage proportional to the working set size rather than the total data store size.
Solution Approach 2:
Instead of caching the entire data store, the system implements partial caching by loading only the pages that are currently needed for processing into memory. This partial action provides sufficient access speed for operational requirements while avoiding the excessive memory consumption of full caching.
3Device complexity
If a simple linear structure is used for the data store, then implementation complexity is reduced, but access efficiency and prefetching capability deteriorate
Solution Approach 1:
The data store is organized into pages with structured metadata, creating a hierarchical structure that enables efficient navigation and prefetching. While this increases implementation complexity compared to a simple linear structure, it dramatically improves access efficiency by allowing the system to load and process related data blocks together.
Solution Approach 2:
The page-based structure enables preliminary action through prefetching, where related pages can be loaded into the cache before they are actually needed for processing. This organizational structure allows the system to anticipate and prepare data in advance, improving access efficiency without requiring complex real-time decisions.
Data Source
AI summary
Techniques for data processing may include: receiving a data chunk and an associated digest; and performing data deduplication processing for the data chunk comprising: determining whether there is an existing entry in a deduplication digest cache for the digest; and responsive to determining there is no existing entry in the deduplication digest cache, performing processing including: determining whether there is an existing entry in a mapping structure for the digest, the mapping structure mapping digests to associated pages of related entries in a deduplication data store; and responsive to determining there is an existing entry in the mapping structure, performing second processing including: obtaining, from the existing entry, a page mapped to the digest; and loading the page of entries from the deduplication data store into the deduplication digest cache. At least some entries of the page may be prefetched and loaded into the deduplication digest cache prior to use.