Partial Recall Deduplication Mechanism for Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication systems face latency and resource consumption issues due to full file recall, especially when dealing with large files, which affects throughput and introduces significant I/O overload.
Innovation Solution
Implementing a partial deduplication mechanism that allows only relevant chunks of a file to be recalled, with tracking information maintained to differentiate between recalled and deduplicated data, enabling efficient management and re-deduplication by scanning only recalled file ranges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full file recall is implemented, then data access is enabled, but latency increases and I/O overload occurs
Solution Approach 1:
The patent segments the file into multiple chunks and selectively recalls only the necessary chunks rather than the entire file. This segmentation allows the system to divide the recall operation into smaller, more manageable units that can be processed in parallel and reduce overall latency.
Solution Approach 2:
The patent extracts and recalls only the specific chunks that are needed for the current operation, leaving the rest of the file data in the deduplicated storage. This selective extraction minimizes the amount of data that needs to be transferred and processed, thereby reducing latency and I/O overhead.
2Reliability
If full file recall is implemented, then data access is enabled, but resource consumption increases
Solution Approach 1:
The patent segments the file into multiple chunks and selectively recalls only the necessary chunks rather than the entire file. This segmentation allows the system to divide the recall operation into smaller, more manageable units that can be processed in parallel and reduce overall latency.
Solution Approach 2:
The patent applies partial action by recalling only the minimum necessary chunks required for the current operation rather than the entire file. This partial recall approach reduces resource consumption by avoiding the overhead of transferring and processing unnecessary data while still enabling the required data access functionality.
3Reliability
If full file recall is implemented, then data access is enabled, but deduplication throughput decreases
Solution Approach 1:
The patent segments the file into multiple chunks and selectively recalls only the necessary chunks rather than the entire file. This segmentation allows the system to divide the recall operation into smaller, more manageable units that can be processed in parallel and reduce overall latency.
Solution Approach 2:
The patent applies partial action by recalling only the minimum necessary chunks required for the current operation rather than the entire file. This partial recall approach reduces resource consumption by avoiding the overhead of transferring and processing unnecessary data while still enabling the required data access functionality.
Data Source
Figure 1
Figure 2~4
Figure 5
AI summary
The subject disclosure is directed towards changing a file from a fully deduplicated state to a partially deduplicated state in which some of the file data is deduplicated in a chunk store, and some is recalled into the file, that is, in the file's storage volume. A partial recall mechanism such as in a file system filter tracks (e.g., via a bitmap in a file reparse point) whether file data is maintained in the chunk store or has been recalled to the file. Data is recalled from the chunk store as needed, and committed (e.g., flushed) to the file. Also described is efficiently returning the file to a fully deduplicated state by using the tracking information to determine which parts of the file are already deduplicated into the chunk store so as to avoid their further deduplication processing.