Online Deduplication Metadata Pruning for Smaller Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The overhead and storage requirements for metadata tables in online deduplication systems are increased due to maintaining entries for data blocks that are not deduplicated, even if they have no references, reducing efficiency.
Innovation Solution
A system that determines a time duration for maintaining metadata entries and removes them if no additional references are made, using predefined times or machine-learning models, ensuring only deduplicated data blocks' metadata is retained.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata entries are maintained for all data blocks in online deduplication, then deduplication accuracy is improved, but metadata table size and overhead increase
Solution Approach 1:
The patent extracts and removes metadata entries corresponding to non-deduplicated data blocks from the metadata table. By identifying and removing unnecessary entries (those without references), the system maintains deduplication accuracy for actual duplicates while reducing metadata table size and overhead.
Solution Approach 2:
The patent applies discarding by removing metadata entries for data blocks that are not deduplicated (have no references). This selective discarding reduces metadata storage requirements while preserving the integrity of the deduplication system by keeping only the necessary entries.
2Loss of information
If metadata entries are maintained for all data blocks, then data block tracking completeness is improved, but system overhead increases
Solution Approach 1:
The patent extracts only the necessary metadata entries (those with references indicating actual deduplication) from the complete set of data block metadata. This extraction maintains tracking completeness for deduplicated blocks while removing overhead from non-deduplicated blocks.
Solution Approach 2:
Instead of maintaining complete metadata for all data blocks, the patent applies partial action by maintaining metadata only for deduplicated data blocks. This partial metadata maintenance reduces system overhead while sufficient for deduplication functionality.
3Reliability
If metadata table includes all data block entries, then deduplication coverage is improved, but storage efficiency deteriorates
Solution Approach 1:
The patent extracts and removes metadata entries for non-deduplicated data blocks from the metadata table. This extraction maintains deduplication coverage for actual duplicates while improving storage efficiency by eliminating unnecessary metadata storage.
Solution Approach 2:
The patent discards metadata entries for data blocks without references, thereby recovering storage space. This selective discarding improves storage efficiency while preserving complete deduplication coverage for actual duplicate blocks.
Data Source
AI summary
A metadata size can be reduced for online deduplication operations. For example, a computing device can generate a metadata entry describing a data block in a metadata table comprising a plurality of metadata entries describing a plurality of data blocks. The computing device can determine a length of time to be associated with the metadata entry for maintaining the metadata entry in the metadata table. The computing device can, subsequent to the length of time passing subsequent to generating the metadata entry, determine the metadata entry indicates a number of references to the data block is below a threshold. The computing device can, in response to determining the number of references is below the threshold, remove the metadata entry from the metadata table.