Deduplication Metadata Scaling With Cloud Detach-Attach Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage appliances face challenges in managing growing data needs due to limited metadata space, leading to high costs for additional storage and maintenance, while cloud storage offers cost-effective scalability but requires efficient integration with deduplicated systems.
Innovation Solution
A metadata-data separated architecture that utilizes cloud storage for data and a detach-attach workflow to manage metadata, allowing scalable cloud capacity without additional local storage, using detach operations to free up space and attach operations to rebuild metadata, optimizing synchronization and parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud storage is used to expand storage capacity, then storage scalability is improved, but metadata space management becomes more complex
Solution Approach 1:
The patent segments metadata into multiple levels (first level metadata stored locally in read-write state, second level metadata stored locally in read-only state, and third level metadata stored in cloud). This segmentation allows the system to scale storage capacity by adding cloud units while managing metadata complexity through hierarchical organization and selective caching strategies.
2Quantity of substance
If additional storage disks are added to storage appliances, then storage capacity is increased, but hardware costs and maintenance expenses increase
Solution Approach 1:
The patent uses cloud storage as a virtual extension of local storage appliances. Instead of physically adding storage disks to appliances, the system creates cloud-based copies/extensions of storage capacity that can be accessed through the existing appliance infrastructure, thereby avoiding additional hardware costs while increasing storage capacity.
3Speed
If all metadata is stored in read-write state, then data access performance is improved, but metadata storage space consumption increases
Solution Approach 1:
The patent implements dynamic metadata management where the state of metadata (read-write or read-only) can change based on access patterns and system conditions. Frequently accessed metadata remains in read-write state for performance, while less frequently accessed metadata is moved to read-only state to conserve space, allowing the system to adapt to changing workloads.
Solution Approach 2:
Different portions of metadata are stored with different characteristics (read-write or read-only) based on their specific access requirements. This local quality differentiation allows the system to optimize both performance and space usage by applying appropriate storage strategies to different metadata segments rather than treating all metadata uniformly.
Data Source
AI summary
Cloud units are maintained in cloud storage. Each cloud unit includes containers including data containers storing segments of files, segment tree containers storing upper-level segments of segment trees representing the files, and cloud containers storing headers from the data and segment tree containers. A header for a data container includes fingerprints identifying the segments of files. A header for a segment tree container includes fingerprints identifying the upper-level segments. A cloud unit is attached to a storage appliance in one of a read-write state or a read-only state by recovering, from cloud storage, to a local storage of the appliance, at least the cloud containers associated with the first cloud unit. While the cloud containers are being recovered, a request is received to access a segment tree of the cloud unit. The request is redirected to cloud storage.