Granular Partial Recall for Deduplicated Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face high latency and resource consumption issues due to full file recall and the need for frequent deduplication, especially in scenarios with small random writes, which limits their ability to support workloads like VDI and Hyper-V scenarios.
Innovation Solution
A granular partial recall mechanism that tracks dirty ranges of file data independently of file-system granularity, allowing for more fine-grained tracking and reducing the need for data recall from the chunk store, combined with a delayed flush mechanism to preserve data integrity and reduce I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full file recall is performed to ensure data consistency and integrity during modifications, then data integrity is maintained, but latency and I/O overhead increase significantly
Solution Approach 1:
The patent segments the file into multiple ranges and implements fine-grained tracking of dirty ranges at the range level rather than recalling the entire file. This allows selective recall of only those ranges that have been modified, significantly reducing latency while maintaining data integrity for the specific ranges that need it.
Solution Approach 2:
The patent applies different quality levels to different parts of the file by implementing fine-grained dirty range tracking. Clean ranges that haven't been modified can remain in the chunk store with full deduplication, while dirty ranges are tracked separately and recalled only when necessary, optimizing the balance between data integrity and performance.
2Ease of operation
If full file recall is performed, then data access is enabled, but resource consumption and I/O overload increase
Solution Approach 1:
The patent divides the file into multiple ranges and tracks dirty ranges individually. This segmentation enables the system to access only the necessary portions of the file rather than recalling the entire file, reducing resource consumption while maintaining data access capability.
Solution Approach 2:
The patent implements partial recall by recalling only the dirty ranges that have been modified rather than performing a full file recall. This partial action approach reduces resource consumption and I/O overhead while still enabling the necessary data access for modifications.
3Measurement precision
If fine-grained dirty range tracking is implemented independent of file-system granularity, then recall precision is improved, but data structure complexity increases
Solution Approach 1:
The patent segments the file into multiple ranges and implements tracking at the range level. This segmentation enables fine-grained precision in tracking dirty ranges while using a structured approach (range-based tracking) that manages complexity through systematic organization rather than ad-hoc tracking.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The subject disclosure is directed towards partially recalling file ranges of deduplicated files based on tracking dirty (write modified) ranges (user writes) in a way that eliminates or minimizes reading and writing already-optimized adjacent data. The granularity of the ranges does not depend on any file-system granularity for tracking ranges. In one aspect, lazy flushing of tracking data that preserves data-integrity and crash-consistency is provided. In one aspect, also described is supporting granular partial recall on an open file while a data deduplication system is optimizing that file.