Block-Level Expiry Scheme for Distributed Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data storage platforms face challenges in maintaining accurate deduplication tracking with low communication overhead, especially in large and active systems, where aggressive space reclamation can impact system performance and storage efficiency.
Innovation Solution
A block-level expiry scheme is implemented in a distributed data storage system, where each deduplicated data block is given an expiry timeframe based on its arrival time and backup frequencies, allowing for proactive aging of stale data blocks and retention of current ones, with a global deduplication virtual disk managing data across storage nodes to optimize deduplication ratios and minimize re-writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If aggressive space reclamation is performed to discard stale data blocks, then storage efficiency is improved, but system performance deteriorates due to increased re-writes of existing deduplicated data
Solution Approach 1:
The patent applies preliminary action by pro-actively aging data blocks before they become stale through the assignment of expiry timeframes. Each deduplicated data block is given an expiry timeframe based on its arrival time and backup frequencies, allowing the system to prepare for and manage space reclamation in advance rather than reacting aggressively when storage space is needed. This prevents performance degradation by avoiding last-minute aggressive reclamation operations.
2Reliability
If data blocks are retained longer to maintain data integrity, then deduplication accuracy is improved, but storage capacity is reduced due to accumulation of stale data blocks
Solution Approach 1:
The patent applies dynamics by making the retention period of data blocks dynamic rather than static. Each deduplicated data block is given an expiry timeframe based on its arrival time and backup frequencies, allowing the system to adapt retention periods to specific data characteristics and backup schedules. This dynamic approach enables the system to maintain data integrity for blocks that need it while automatically discarding blocks that are no longer needed, optimizing both reliability and storage capacity utilization.
3Measurement precision
If block-level expiry tracking is implemented to manage individual data blocks, then space reclamation precision is improved, but communication overhead increases among distributed components
Solution Approach 1:
The patent applies segmentation by dividing the expiry tracking mechanism into distributed components that operate independently at each storage node. Each node maintains its own expiry timeframe information for local deduplicated data blocks, eliminating the need for continuous centralized coordination. This segmentation enables precise block-level space reclamation while minimizing network communication overhead, as each node can make autonomous decisions about expiring local blocks based on their expiry timeframes.
Data Source
AI summary
Disclosed deduplication techniques at a distributed data storage system guarantee that space reclamation will not affect deduplicated data integrity even without perfect synchronization between components. By understanding certain “behavioral” characteristics and schedule cadences of backup operations that generate backup copies received at the distributed data storage system, data blocks that are not re-written by subsequent backup copies are pro-actively aged, while promoting continued retention of data blocks that are re-written. An expiry scheme operates with block-level granularity. Each unique deduplicated data block is given an expiry timeframe based on the block's arrival time at the distributed data storage system (i.e., when a backup copy supplies the block) and further based on backup frequencies of the various virtual disks referencing a unique system-wide identifier of the block, which is based on the block's hash value. Communications between components are kept to an as-needed basis. Cloud-based and multi-cloud configurations are disclosed.