Upstream Server Load Management via Pushback Signals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems without a global load manager, upstream servers may inadvertently overload downstream servers due to unmanaged replication requests, leading to potential system instability and reduced performance.
Innovation Solution
Implementing a method for the upstream server to monitor and adjust the transaction rate limit based on the downstream server's maximum transaction processing rate and pushback signals, ensuring that transactions are issued at a rate that does not exceed the downstream server's capacity, thereby preventing overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If upstream servers issue replication requests without a global load manager, then the distributed storage system achieves simplicity and scalability, but downstream servers may be overloaded
Solution Approach 1:
The downstream server autonomously manages its own load by monitoring its transaction processing rate and sending pushback signals to upstream servers when approaching capacity thresholds. This self-service mechanism eliminates the need for a global load manager while preventing overload conditions.
Solution Approach 2:
The system implements feedback through pushback signals sent from downstream servers to upstream servers. When the downstream server's transaction processing rate approaches the threshold, it sends pushback signals to reduce the replication request rate, creating a closed-loop control system that prevents overload.
2Reliability
If upstream servers increase replication request rate to improve data availability, then system reliability improves, but downstream server capacity is exceeded
Solution Approach 1:
The system dynamically adjusts the replication request rate based on real-time monitoring of the downstream server's transaction processing rate. The threshold for pushback signals can be configured dynamically, allowing the system to adapt to changing load conditions while maintaining data availability.
Solution Approach 2:
The feedback mechanism through pushback signals allows the system to respond to capacity constraints by reducing replication request rates when necessary, preventing downstream server overload while maintaining data availability through controlled replication.
3Reliability
If the transaction rate limit is strictly enforced, then downstream server overload is prevented, but replication speed is reduced
Solution Approach 1:
The system uses periodic monitoring of the downstream server's transaction processing rate and sends pushback signals only when necessary. This periodic action allows high-speed replication to proceed normally under acceptable load conditions while preventing overload only when needed.
Solution Approach 2:
The system changes the effective transaction rate limit parameter dynamically based on downstream server conditions. When the transaction processing rate approaches the threshold, the limit is enforced through pushback signals; when below the threshold, replication can proceed at higher speeds.
Data Source
AI summary
The various embodiments described herein include methods, systems, and devices for managing load in a distributed storage system. In one aspect, a method is performed at a first instance server in the distributed storage system, the first instance server having memory and at least one processor coupled to the memory. The method includes: (i) issuing a first plurality of requests to a second instance server; (ii) obtaining one or more messages from the second instance server in response to the first plurality of requests, the messages indicating a utilization rate of the second instance server; (iii) determining a transaction rate limit for the second instance server based on the utilization rate of the second instance server; and (iv) issuing a second plurality of requests to the second instance server, where the second plurality of requests are issued at a rate no greater than the transaction rate limit.