Storage System Resource Allocation via Compute Core Sharding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Load balancing in multiple-node storage systems leads to increased network load and latency for client computers due to rerouting of I/O requests, which is inefficient and affects response times.
Innovation Solution
Implementing a method that logically shards compute workload into units called shards, dynamically assigning sharded and non-sharded tasks to compute cores, allowing for shard relocation to balance workload and minimize latency, while maintaining minimal disruption to ongoing operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If I/O requests are rerouted to the assigned compute node based on address ranges, then load balancing is achieved, but network load increases and latency increases
Solution Approach 1:
The system performs preliminary action by having the receiving compute node check whether it is the assigned compute node for the requested address range before processing the I/O request. This upfront verification prevents unnecessary rerouting operations and reduces latency while maintaining load balancing effectiveness.
Solution Approach 2:
The system implements feedback by using the response path to provide information about assignment ownership. When a compute node receives a response from storage, it forwards the response to the original requesting entity, creating a feedback loop that enables the system to learn and optimize routing decisions while maintaining load balance.
2Productivity
If I/O requests are rerouted to maintain load balance, then compute node load is balanced, but network load increases
Solution Approach 1:
The receiving compute node performs a preliminary check to determine if it is the assigned compute node for the requested address range before proceeding with request handling. This preliminary action eliminates unnecessary rerouting and reduces network load while preserving load balancing benefits.
Solution Approach 2:
The system extracts and eliminates the harmful rerouting operation by implementing direct handling at the receiving node when possible. By taking out the unnecessary rerouting step, the system reduces network traffic and energy consumption while maintaining effective load distribution.
3Loss of time
If responses are sent via the receiving compute node, then client response time is improved, but more rerouting operations are required
Solution Approach 1:
The system performs preliminary verification at the receiving compute node to determine if it is the assigned node for the requested address range. This upfront check enables the receiving node to handle the request directly without rerouting, reducing both client response time and rerouting complexity simultaneously.
Solution Approach 2:
Instead of always rerouting requests to the assigned node and then responding through the original receiver, the system inverts the approach by allowing the receiving node to handle the request directly if it is the assigned node. This inversion eliminates unnecessary rerouting while maintaining fast client response times.
Data Source
AI summary
A method for resource allocation in a storage system, the method may include determining to reduce a load imposed on a compute core of the storage system, wherein the compute core is responsible for executing (a) one or more background task shards, and (b) at least one non-sharded task for responding to an input/output (I/O) request sent to the storage system; and reducing the load imposed on the compute core by reallocating at least one background task shard of the one or more background task shards to another compute core of the storage system.