Snapshot Changed Blocks Metadata Querying Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large block sizes in block storage services lead to inefficient resource usage when identifying changed blocks, as each block requires a separate API call, resulting in excessive network traffic and resource consumption.
Innovation Solution
Organizing blocks into clusters, where each cluster includes a predefined number of blocks, allowing for efficient iteration through clusters to identify changed blocks, reducing the number of blocks that need to be processed by focusing on changed clusters and their constituent blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If smaller block sizes are used to improve storage efficiency and alignment with physical media, then the number of API calls required to identify changed blocks increases significantly, resulting in excessive network traffic and resource consumption
Solution Approach 1:
The patent divides the storage volume into clusters of blocks, where each cluster contains multiple smaller blocks (e.g., 8192 blocks per cluster). This segmentation allows the system to use smaller block sizes for efficient storage while grouping them into larger clusters for querying purposes, thereby reducing the number of API calls needed to identify changed blocks.
Solution Approach 2:
The patent introduces cluster metadata as an intermediary layer between the storage volume and the querying interface. This metadata structure provides a hierarchical view that maps clusters to their constituent blocks, enabling efficient navigation and reducing the number of individual block queries required while maintaining support for smaller block sizes.
2Productivity
If larger block sizes are used to reduce the number of API calls, then storage efficiency decreases and alignment with physical media block sizes (e.g., 4 KB) is lost
Solution Approach 1:
The patent segments the storage volume into a hierarchical structure where clusters contain multiple smaller blocks. This allows the system to maintain smaller block sizes (e.g., 4 KB) that align with physical media while organizing them into larger clusters for efficient querying, thus resolving the contradiction between block size and API call efficiency.
Solution Approach 2:
The patent introduces a new dimensional organization by creating clusters that group multiple blocks together. This adds a hierarchical dimension to the storage structure, allowing efficient querying at the cluster level while maintaining fine-grained control at the block level, thereby achieving both storage efficiency and API call efficiency.
3Measurement precision
If traditional block-by-block querying is used, then each block requires a separate API call, resulting in overwhelming network traffic when dealing with large storage volumes
Solution Approach 1:
The patent segments the querying process into two levels: first querying at the cluster level to identify changed clusters, then querying only the blocks within those changed clusters. This segmentation dramatically reduces the number of API calls required while maintaining accurate block-level change detection, thereby reducing network traffic without sacrificing measurement precision.
Solution Approach 2:
The patent performs preliminary action by first identifying changed clusters before querying individual blocks. This preliminary filtering step eliminates the need to query unchanged blocks, significantly reducing network traffic while ensuring that all changed blocks are eventually detected with the same accuracy as traditional block-by-block querying.
Data Source
AI summary
Techniques for optimizing snapshot changed blocks metadata querying are described. A method of optimizing snapshot changed blocks metadata querying comprises receiving a request to identify one or more changed blocks of a block storage volume in a storage service of a provider network, identifying at least one changed cluster of blocks of the storage volume, the at least one cluster of blocks comprising a plurality of blocks of the storage volume, and identifying at least one changed block from the plurality of blocks associated with the at least one changed cluster.