Deduplication Metadata Segmentation for Unreferenced Block Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deduplication systems face inefficiencies in managing metadata for data blocks, particularly in identifying and handling unreferenced data blocks, which can lead to unnecessary storage and resource utilization issues.
Innovation Solution
A method and system for maintaining metadata in a deduplication system that includes identifying unreferenced data blocks, updating file metadata with their references, and optionally moving them to a secondary storage device, while retaining metadata for potential future use in deduplication operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is maintained for all data blocks including unreferenced blocks, then deduplication effectiveness is improved, but storage overhead and resource utilization deteriorate
Solution Approach 1:
The patent segments metadata management into two distinct categories: active metadata for referenced data blocks and inactive metadata for unreferenced data blocks. This segmentation allows the system to maintain metadata for deduplication effectiveness while separately managing unreferenced blocks to control storage overhead. The metadata is divided into active metadata structures (for current file references) and inactive metadata structures (for potential future references), enabling selective retention and management.
Solution Approach 2:
The patent implements preliminary action by proactively identifying and separating unreferenced data blocks before they consume excessive storage resources. The system performs preliminary metadata management by moving unreferenced blocks to a separate inactive metadata structure, preparing them for potential future recovery or deletion without interfering with current active deduplication operations. This preliminary separation prevents storage overhead accumulation while maintaining deduplication capability.
2Quantity of substance
If unreferenced data block metadata is immediately removed, then storage overhead is reduced, but future deduplication opportunities are lost
Solution Approach 1:
The patent introduces an intermediary inactive metadata structure that serves as a buffer between active metadata and complete deletion. This intermediary structure holds unreferenced data block metadata, allowing the system to reduce storage overhead by removing active metadata entries while preserving the ability to recover or utilize these blocks for future deduplication. The inactive metadata acts as a mediator that balances storage reduction with future deduplication potential.
Solution Approach 2:
The patent implements discarding and recovering by moving unreferenced data block metadata from active structures to inactive structures, effectively discarding immediate active references while preserving recovery potential. The system can later recover these blocks from inactive metadata if needed, enabling future deduplication opportunities. This process reduces current storage overhead while maintaining the capability to recover and utilize unreferenced blocks when appropriate.
3Speed
If all data blocks are retained in active metadata, then access speed is improved, but device complexity increases
Solution Approach 1:
The patent segments the metadata management system into distinct active and inactive components, each with specific access patterns and management procedures. Active metadata structures optimize for fast access during current file operations, while inactive metadata structures handle unreferenced blocks with different access characteristics. This segmentation reduces overall device complexity by creating specialized, manageable components rather than a monolithic metadata system.
Solution Approach 2:
The patent extracts unreferenced data block metadata from the active metadata structure, removing complexity from the primary access path. By taking out inactive metadata entries and placing them in separate inactive structures, the system simplifies active metadata management and access operations while maintaining the ability to handle unreferenced blocks when needed. This extraction reduces the complexity burden on the active metadata system.
Data Source
AI summary
Provided are a method, system, and article of manufacture for managing metadata for data blocks used in a deduplication system. File metadata is maintained for files having data blocks in a computer readable device. Data block metadata is maintained for each data block in the computer readable device. The data block metadata for one data block includes a data block reference and content identifier identifying content of the data block. The file metadata for each file includes the data block reference to each data block in the file. A determination is made of an unreferenced data block in the computer readable device that has become unreferenced. Indication is made that the data block metadata for the determined unreferenced data block as unreferenced metadata. The data block reference of the unreferenced metadata is maintained in the computer readable device in response to determining that a includes the data block indicated in the unreferenced metadata.