Hierarchical Workload Allocation for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face bottlenecks due to uneven workload distribution, leading to inefficient resource utilization and metadata waste, especially when scaling out, as existing methods either underutilize or overutilize shards and workload allocation units.
Innovation Solution
A hierarchical workload allocation method that dynamically reallocates first and second type shards among workload allocation units, maintaining association with compute cores while optimizing load balancing, allowing for efficient scaling without disrupting ongoing tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If workload is statically allocated to shards, then system structure is simple, but resource utilization becomes inefficient when scaling out
Solution Approach 1:
The patent implements dynamic workload allocation where the system automatically reassigns shards to compute cores based on current system state. When compute nodes are added or removed, the workload allocation unit dynamically recalculates and redistributes shards to maintain optimal balance, transforming the static allocation model into an adaptive dynamic system that responds to scaling events.
Solution Approach 2:
The system changes the allocation parameters (which compute core handles which shard) based on system configuration. The workload allocation unit monitors compute node availability and adjusts the mapping between compute cores and shards, changing the distribution parameters to optimize resource utilization without requiring fixed static assignments.
2Productivity
If shards are reallocated during scale-out, then resource utilization improves, but ongoing storage operations may be disrupted
Solution Approach 1:
The workload allocation unit performs preliminary assessment before reallocation by identifying which compute nodes are newly added or removed. It plans the shard redistribution in advance, determining the optimal target allocation before actually moving shards, thereby preparing the system to minimize disruption to ongoing operations.
Solution Approach 2:
The system performs rapid shard reassignment during scale-out events by skipping unnecessary intermediate steps. The workload allocation unit directly reallocates shards from underutilized compute cores to newly added compute cores in a single coordinated operation, reducing the time window during which operations could be disrupted.
3Quantity of substance
If new compute nodes are added, then system capacity increases, but workload balancing becomes more complex
Solution Approach 1:
The workload allocation unit implements self-service automation that autonomously detects when compute nodes are added to the cluster and automatically performs the complex task of redistributing shards. The system serves itself by monitoring its own state and executing rebalancing operations without external intervention, handling the increased complexity of workload balancing across more nodes through automated decision-making.
4Device complexity
If workload is concentrated on fewer nodes, then metadata management is simpler, but system bottlenecks increase
Solution Approach 1:
The system segments the workload by dividing shards and distributing them across multiple compute cores. The workload allocation unit creates a segmented distribution pattern where no single compute node holds all shards, thereby segmenting the metadata management burden and preventing bottlenecks while maintaining manageable complexity through systematic distribution rules.
Data Source
AI summary
A method for hierarchical workload allocation in a storage system, the method may include determining to reallocate a compute workload of a current compute core of the storage system; wherein the current compute core is responsible for executing a workload allocation unit that comprises one or more first type shards; and reallocating the compute workload by (a) maintaining the responsibility of the current compute core for executing the workload allocation unit, and (b) reallocating at least one first type shard of the one or more first type shards to a new workload allocation unit that is allocated to a new compute core of new compute cores.