Snapshot Chain Sharding for Parallel VM Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in efficiently managing and restoring different point in time versions of virtual machines due to large file sizes, which can lead to disk space issues and impaired performance in high storage utilization environments, especially when consolidating or reversing snapshot chains.
Innovation Solution
The system splits snapshot chains into parallel shards or sub-chains with file sizes less than a threshold, allowing for independent parallel processing and management, thereby improving read and write performance and reducing disk space requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If snapshot chains are stored as large files, then storage capacity is maximized, but read and write performance deteriorates and disk space management becomes difficult
Solution Approach 1:
The patent divides snapshot chains into smaller shards or sub-chains with file sizes below a threshold. This segmentation allows the system to maintain storage capacity while improving read and write performance, as smaller files can be processed more efficiently and enable parallel operations during consolidation and reversal operations.
2Quantity of substance
If snapshot chains are stored as large files, then storage efficiency is improved, but disk space requirements increase and management complexity increases in high utilization environments
Solution Approach 1:
By segmenting snapshot chains into smaller shards, the system maintains storage efficiency through consolidation while reducing the complexity of disk space management. Smaller shard files are easier to manage, move, and consolidate, particularly in high storage utilization environments where space optimization is critical.
Solution Approach 2:
The patent enables merging of smaller shards back into consolidated snapshot chains when space permits, optimizing storage efficiency. This combination of segmentation and merging allows flexible disk space management that adapts to changing storage conditions.
3Productivity
If snapshot chains are split into smaller shards, then read and write performance improves and disk space management becomes easier, but system complexity increases
Solution Approach 1:
The system automatically manages the segmentation of snapshot chains into shards, handling the complexity of file division, storage, and retrieval. This automation reduces the perceived system complexity while maintaining the performance benefits of smaller file sizes.
Solution Approach 2:
The patent introduces an intermediary layer that manages the shard sub-chain structure, handling the complexity of parallel processing and consolidation operations. This intermediary abstraction allows the system to benefit from improved performance while shielding users from the underlying complexity.
Data Source
AI summary
Methods and systems for backing up and restoring different point in time versions of a virtual machine, an application, a database, or an electronic file using independently managed snapshot chains are described. In some cases, different point in time versions of a virtual machine may be captured and stored using one or more snapshot chains. Each snapshot chain may correspond with a base image (e.g., a full image snapshot) and one or more incremental files (e.g., two forward incremental files) that derive from the base image. A snapshot chain may be split into a plurality of shards or sub-chains such that the width of each shard or the maximum data size of the files in each shard is less than a threshold data size (e.g., is less than 1 TB). Data operations (e.g., consolidation and reverse operations) may be performed on the individual shards or sub-chains in parallel.