Snapshot Metadata Arrangement for Cloud Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management strategies face challenges in efficiently creating and restoring snapshots across different data management paradigms, particularly in cloud-integrated storage systems, where snapshot metadata and data management are not optimized for efficient retrieval and retention, leading to potential violations of service level objectives and increased storage costs.
Innovation Solution
A storage appliance that arranges snapshot metadata and data into key-value stores, using locally generated snapshot identifiers to facilitate efficient snapshot creation, restoration, and reclamation, while maintaining metadata for both cached and evicted snapshots, and employing synthetic baseline snapshots to reduce stable data eviction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If snapshot metadata is stored in traditional database structures, then data management is straightforward, but retrieval efficiency and restoration speed are reduced
Solution Approach 1:
The patent segments snapshot metadata into distinct key-value stores separated by snapshot identifiers. Each key-value store contains only the metadata relevant to a specific snapshot, allowing parallel and independent retrieval operations. This segmentation eliminates the need to scan through all metadata to restore a particular snapshot, dramatically improving restoration speed while maintaining manageable complexity through structured organization.
Solution Approach 2:
The patent introduces a new dimensional organization to metadata storage by creating separate key-value stores for each snapshot rather than using a single flat database structure. This dimensional change allows the system to access metadata in O(1) time complexity by directly accessing the appropriate key-value store, transforming the retrieval operation from a linear search problem to a direct access problem.
2Speed
If all snapshots are retained in cache, then restoration speed is improved, but storage costs and memory usage increase
Solution Approach 1:
The patent applies local quality by maintaining metadata in cache for all snapshots while allowing the actual snapshot data to be distributed across different storage tiers. Frequently accessed snapshots can have their data kept in faster storage, while less frequently accessed data can be moved to slower, cheaper storage. This localized optimization allows the system to maintain high restoration speeds for critical data without proportionally increasing overall storage capacity requirements.
Solution Approach 2:
The patent implements a nested storage architecture where metadata is stored in an inner layer (cache/memory) and snapshot data is stored in an outer layer (distributed storage system). The key-value stores act as an index layer that nestles between the cache and the full snapshot data, allowing the system to quickly locate and retrieve only the necessary data portions without requiring all snapshot data to reside in cache simultaneously.
3Quantity of substance
If incremental snapshots are used to reduce storage usage, then storage efficiency improves, but restoration complexity and time increase
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing change vectors that map incremental snapshot changes to their corresponding baseline snapshot data locations. When restoration is needed, the system can quickly apply these pre-computed change vectors to reconstruct the snapshot without having to replay entire incremental backup streams. This preliminary preparation significantly reduces restoration time while maintaining the storage efficiency benefits of incremental snapshots.
Solution Approach 2:
The patent introduces change vectors as an intermediary data structure that bridges incremental snapshots and baseline snapshots. These change vectors contain metadata about what data changed, where it changed, and how it changed, allowing the restoration process to efficiently navigate between incremental and baseline data without requiring complex sequential processing of all incremental snapshots.
4Ease of operation
If cloud service provider resources are used, then infrastructure maintenance is reduced, but visibility and control over resources decrease
Solution Approach 1:
The patent positions the key-value store metadata system as an intermediary layer between the application and cloud storage resources. This metadata layer provides a standardized interface for managing snapshots while maintaining visibility into the state and location of data. The system can track metadata about cloud resources without requiring direct visibility into the physical infrastructure, balancing ease of cloud resource management with sufficient information visibility for data protection operations.
Data Source
AI summary
A storage appliance arranges snapshot data and snapshot metadata into different structures, and arranges the snapshot metadata to facilitate efficient snapshot manipulation, which may be for snapshot management or snapshot restore. The storage appliance receives snapshots according to a forever incremental configuration and arranges snapshot metadata into different types of records. The storage appliance stores these records in key-value stores maintained for each defined data collection (e.g., volume). The storage appliance arranges the snapshot metadata into records for inode information, records for directory information, and records that map source descriptors of data blocks to snapshot file descriptors. The storage appliance uses a locally generated snapshot identifier as a key prefix for the records to conform to a sort constrain of the key-value store, which allows the efficiency of the key-value store to be leveraged. The snapshot metadata arrangement facilitates efficient snapshot restore, file restore, and snapshot reclamation.