Extent-Based Deduplication for Storage Space Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication algorithms in network-based storage systems impact performance and do not maximize space efficiency, as they require processing data inline and do not deduplicate metadata associated with active file systems and snapshots.
Innovation Solution
Implementing deduplication in an extent-based architecture, where a log data container is used to identify and manage duplicate data entries, allowing for block sharing and freeing of donor extents when reference counts meet predetermined values, and performing deduplication after data has been written, enabling metadata deduplication for improved space efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If deduplication is performed inline during data writing, then duplicate data can be identified and removed, but system performance deteriorates due to computational overhead
Solution Approach 1:
The patent performs deduplication operations after data has been written to storage, rather than during the write operation. This preliminary action (completing the write first) separates the deduplication computational overhead from the critical write path, thus maintaining system performance while still achieving duplicate data removal and space efficiency
2Loss of substance
If traditional block-based deduplication is used, then duplicate data blocks can be identified, but metadata associated with active file systems and snapshots cannot be deduplicated, limiting space efficiency
Solution Approach 1:
The patent extends deduplication capabilities to work with extent-based architecture that can handle both data blocks and metadata structures uniformly. The system can identify and deduplicate duplicates across different data types including file system metadata and snapshot metadata, making the deduplication mechanism universal and applicable to all storage objects rather than being limited to just data blocks
3Loss of substance
If inline deduplication processing is implemented, then duplicate data can be eliminated in real-time, but computational overhead increases and impacts write performance
Solution Approach 1:
The patent performs deduplication operations after data has been written to storage, rather than during the write operation. This preliminary action (completing the write first) separates the deduplication computational overhead from the critical write path, thus maintaining system performance while still achieving duplicate data removal and space efficiency
Data Source
AI summary
A request is received to remove duplicate data. A log data container associated with a storage volume in a storage server is accessed. The log data container includes a plurality of entries. Each entry is identified by an extent identifier in a data structures stored in a volume associated with the storage server. For each entry in the log data container, a determination is made if the entry matches another entry in the log data container. If the entry matches another entry in the log data container, a determination is made of a donor extent and a recipient extent. If an external reference count associated with the recipient extent equals a first predetermined value, block sharing is performed for the donor extent and the recipient extent. A determination is made if the reference count of the donor extent equals a second predetermined value. If the reference count of the donor extent equals the second predetermined value, the donor extent is freed.