Correlated Volume Placement in Distributed Block Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing block storage services face challenges in efficiently handling large, interrelated requests for storage volumes, leading to potential failures and suboptimal resource utilization due to oversubscription and distributed system complexities, where individual treatment of requests can result in undesirable outcomes such as co-location of correlated volumes and resource overload.
Innovation Solution
Implementing a context-aware placement algorithm that considers the interrelated nature of volume requests, using contextual information to spread volumes across multiple hosts and injecting randomness into placement decisions to reduce the probability of oversubscription and optimize resource usage, thereby ensuring efficient distribution of loads and maintaining performance guarantees.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If individual volume requests are handled independently in a distributed block storage service, then processing speed and scalability are improved, but the likelihood of co-location of correlated volumes increases leading to resource overload and request failures
Solution Approach 1:
The placement algorithm incorporates feedback about the interrelated nature of volume requests and the current distribution state across hosts. By monitoring which hosts already have correlated volumes and adjusting placement decisions accordingly, the system prevents resource overload while maintaining high processing speed for large requests
Solution Approach 2:
The system dynamically changes placement parameters based on request characteristics. When correlated volumes are detected, the algorithm modifies placement probabilities and host selection criteria to distribute volumes across different hosts, preventing co-location issues while maintaining efficient processing
2Productivity
If volumes are spread across multiple hosts to avoid oversubscription, then resource utilization is optimized, but the complexity of placement decisions increases
Solution Approach 1:
The system performs preliminary analysis of volume requests to identify correlated groups before placement decisions are made. By pre-processing requests to determine interrelationships and pre-evaluating suitable hosts, the algorithm reduces placement complexity while achieving optimal resource distribution across hosts
Solution Approach 2:
The placement problem is segmented into independent sub-tasks that can be processed in parallel. The algorithm divides the distributed placement decision into manageable components, allowing complex resource optimization to be achieved without overwhelming system complexity
3Reliability
If randomness is injected into placement decisions to reduce oversubscription probability, then service reliability is improved, but placement precision decreases
Solution Approach 1:
The system applies randomness partially rather than completely. Placement decisions use a hybrid approach where deterministic rules provide baseline precision and randomization is applied selectively to reduce oversubscription probability, achieving both reliability and acceptable precision
Data Source
AI summary
Systems and methods are disclosed for handling requests to create multiple volumes with an expected usage correlation on a block storage service. Rather than handling each request to create a volume independently, embodiments described herein can handle the request in bulk. In one embodiment, the service allows for oversubscription of use on hosts, as well as parallelization of placement decisions, by distributing requests among a set of parallelized placement engines. Each engine can distribute its subset of volumes at least partly randomly among a candidate set of volumes, with the size of the candidate set selected based on a total number of volumes. This distribution mechanism can ensure distribution of volumes without requiring centralized placement of the volumes.