Delayed Deduplication via Precalculated Hash Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication schemes face inefficiencies in resource consumption and storage efficiency, as background deduplication is disk-intensive and slow, while inline deduplication is costly and may stop working under high storage demands, often requiring large memory caches that are not available.
Innovation Solution
A method where digests of newly-ingested data blocks are stored in a log with information about the blocks, allowing for a delayed deduplication process that proceeds based on log content, reducing the need for memory loading and deferring resource-intensive tasks until resources are available, with optional inline deduplication for blocks found in the cache.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If background deduplication is used, then storage efficiency is improved, but processing speed and resource consumption worsen
Solution Approach 1:
The system performs preliminary actions by computing digests of data blocks and storing them in a log immediately upon ingestion. This preliminary processing enables the delayed deduplication process to proceed efficiently without needing to re-compute hashes or load entire blocks into memory, thus improving both storage efficiency and processing speed.
Solution Approach 2:
The deduplication process is segmented into two phases: (1) inline phase where digests are computed and logged immediately upon data ingestion, and (2) delayed phase where the log is processed to perform actual deduplication operations. This segmentation allows the system to achieve thorough deduplication without the performance penalty of loading entire blocks into memory during the critical path.
2Productivity
If inline deduplication is used, then processing speed is improved, but memory requirements worsen
Solution Approach 1:
The system extracts only the essential information (digests and metadata) from data blocks and stores them in a log, rather than loading entire blocks into memory for inline deduplication. This extraction approach enables fast processing by working with small digest values while maintaining the ability to perform thorough deduplication, thus reducing memory requirements while preserving processing speed.
3Reliability
If full inline deduplication is implemented, then storage efficiency is improved, but device complexity and resource consumption worsen
Solution Approach 1:
The system dynamically adapts its behavior based on available resources and operational conditions. It performs inline deduplication when resources permit and falls back to delayed deduplication via the log when resources are constrained. This dynamic approach enables the system to achieve high storage efficiency while adapting resource consumption to actual system capabilities, reducing overall device complexity.
Data Source
AI summary
A method of performing deduplication by a computing device is provided. The method includes (a) as data is received by the computing device into blocks as part of write requests, creating an entry in a log for each of the blocks, each entry including information about that respective block and a digest computed from that respective block; and (b) after accumulating multiple entries in the log, processing the log for delayed deduplication, the processing including (i) retrieving digests from the log, (ii) performing lookups within a deduplication table of the retrieved digests, and (iii) performing deduplication operations based on the lookups using the information about blocks included within the log. An apparatus, system, and computer program product for performing a similar method are also provided.