Deduplication Efficiency via Write Activity Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies due to the creation of stale digest entries in index tables, leading to increased storage and memory consumption, and performance degradation, especially when dealing with frequently updated data blocks like file system metadata, database transaction logs, and new email data.
Innovation Solution
The proposed method improves deduplication efficiency by skipping data blocks for deduplication processing based on their write activity, reducing the incidence of stale digest entries and optimizing resource usage by prioritizing stable data blocks for deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data blocks are deduplicated without evaluating write activity, then deduplication coverage is maximized, but stale digest entries increase causing memory consumption and performance degradation
Solution Approach 1:
The system performs preliminary evaluation of metadata to determine write activity status before attempting deduplication. This preliminary action identifies data blocks that are unlikely to become stale, preventing unnecessary deduplication processing and avoiding creation of stale digest entries in the first place
Solution Approach 2:
Instead of attempting to deduplicate all data blocks equally, the system applies deduplication selectively only to blocks with low write activity. This partial action approach focuses resources on blocks that will remain valid longer, reducing the creation of stale entries while maintaining effective deduplication coverage
2Loss of substance
If all data blocks are processed for deduplication, then storage efficiency is improved, but index table size increases consuming more storage resources
Solution Approach 1:
The system extracts and evaluates metadata information from data blocks to determine write activity characteristics. This extraction process enables identification of blocks with low write activity that are suitable for deduplication, while excluding blocks that would generate stale entries and unnecessarily increase index table size
Solution Approach 2:
The system applies different deduplication processing strategies based on local characteristics of individual data blocks. Blocks with low write activity receive full deduplication processing, while blocks with high write activity are excluded, creating a non-uniform but optimized deduplication approach that reduces index table growth
3Productivity
If deduplication is applied to frequently updated data blocks, then immediate deduplication ratio is improved, but performance degradation occurs due to continuous reduplication
Solution Approach 1:
The system performs preliminary evaluation of write activity metadata to identify and exclude frequently updated data blocks before they can undergo deduplication processing. This preliminary anti-action prevents the creation of stale digest entries that would require continuous reduplication and associated performance degradation
Solution Approach 2:
The system converts the potentially harmful effect of frequent writes (which would cause continuous reduplication) into a beneficial filtering criterion. By using write activity metadata as a selection criterion, the system identifies blocks to exclude from deduplication, thereby eliminating the performance degradation problem while maintaining deduplication effectiveness on stable blocks
Data Source
AI summary
A method is used in improving deduplication efficiency. Metadata of a data object is evaluated for determining write activity of the data object. Based on the write activity, deduplicating technique is applied to the data object.