Metadata Delta Log Deduplication Using Blind Reference Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in flushing write operations due to flushing bottlenecks and resource depletion, leading to degraded performance and limited write operation request bandwidth.
Innovation Solution
Implementing a method for data deduplication that performs blind increment reference counting (incred) during destaging of write operations without loading metadata pages into cache, optimizing the flushing process and reducing the need for cache access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata pages are loaded into cache during destaging of write operations, then deduplication processing can be performed accurately, but cache resources are depleted and flushing bottlenecks occur leading to degraded performance
Solution Approach 1:
The patent applies preliminary action by recording reference count updates in the metadata log during the destaging process, before the actual metadata pages need to be accessed. This allows the reference counts to be prepared in advance, so when deduplication processing occurs, the information is already available without requiring cache loading, thus maintaining reliability while avoiding performance degradation
Solution Approach 2:
The patent introduces an intermediary mechanism by using a metadata log that stores reference count information separately from the main metadata pages. This intermediary log allows the system to track and manage reference counts without loading the entire metadata pages into cache, resolving the contradiction between accurate deduplication and cache resource consumption
2Measurement precision
If reference counts are updated by reading current values from metadata pages during destaging, then accurate reference counting is maintained, but additional cache access is required increasing I/O latency
Solution Approach 1:
The system performs preliminary action by capturing and recording reference count update information in the metadata log during the destaging phase, before any metadata page access is needed. This preliminary recording ensures that when reference count verification is required, the data is already available in memory, maintaining measurement precision while eliminating the time penalty of additional I/O operations
Solution Approach 2:
The patent uses copying by creating a copy of the reference count update information and storing it in the metadata log. This copy contains all necessary information about reference count changes, allowing the system to maintain accurate reference counting without needing to access the original metadata pages, thus reducing I/O latency while preserving measurement precision
3Productivity
If blind increment reference counting is implemented without loading metadata pages, then cache access is reduced and flushing efficiency improves, but additional processing is deferred to metadata log destaging
Solution Approach 1:
The patent applies the taking out principle by extracting the reference count update information from the metadata pages and storing it separately in the metadata log during destaging. This extraction allows blind increment reference counting to proceed without loading metadata pages into cache, improving flushing efficiency. The extracted information is then used during metadata log destaging to update the actual reference counts, managing the additional processing in a controlled manner
Data Source
AI summary
In at least one embodiment, destaging a write of a user data (UD) log may include: performing deduplication that determines content written by the write to a logical address LA is a duplicate of existing content; and establishing mapping information of a chain of metadata (MD) pages mapping LA to a physical address PA storing the existing content, wherein the chain includes a MD leaf page and a VLB (virtual layer block) page. An update of a MD log can update an indirect pointer (IDP) field of a MD leaf entry of the MID leaf page to reference a VLB entry of the VLB page where the VLB entry further includes PA. A second update of the MD log can increment a reference count of the VLB entry. The second update can be recorded blindly in the MD log without reading or accessing the reference count during destaging the write.