Distributed Service Throughput Control via Multiple Algorithms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed service environments face challenges in managing node backlogs and smooth throughput reduction when a node is brought back online or experiences a large backlog, leading to an unpleasant user experience due to sudden drops in service availability.
Innovation Solution
Implementing multiple algorithms within the distributed service to dynamically adjust the rate of work acceptance, switching between a first algorithm for normal operational mode and a second algorithm for recovery mode, which smoothes out throughput reductions and ensures consistent service availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the distributed service stops receipt of user requests to allow a backlogged storage node to catch up, then the backlog of the storage node is reduced, but the user experience deteriorates due to service unavailability
Solution Approach 1:
The system dynamically adjusts the throughput reduction applied to incoming requests based on the backlog severity of recovering nodes. Instead of a static stop-all-requests approach, the system applies dynamic throttling that adapts to current system state, allowing partial service continuity while still enabling backlog recovery.
Solution Approach 2:
The system changes the parameter of request acceptance rate from 100% (normal operation) to a reduced rate (during recovery), controlled by the gentle reduction factor. This parameter adjustment allows the system to balance between clearing backlogs and maintaining service availability, rather than making binary stop/go decisions.
2Productivity
If the distributed service abruptly stops incoming requests to clear node backlogs, then the node backlog is cleared faster, but the throughput reduction becomes jarring to users
Solution Approach 1:
The system applies a gentle reduction factor as a cushioning mechanism that softens the impact of throughput reduction on users. Instead of abrupt stops, the system pre-applies a gradual reduction that cushions the transition, making the backlog clearance process imperceptible or minimally noticeable to users.
Solution Approach 2:
The system periodically evaluates node backlog states and adjusts the gentle reduction factor accordingly, rather than applying a fixed reduction. This periodic adjustment allows the system to clear backlogs in a staged manner, maintaining smooth user experience while progressively reducing backlogs over time.
3Device complexity
If the distributed service uses a single algorithm for determining work acceptance rate, then the system complexity is reduced, but the ability to handle different operational modes (normal and recovery) is compromised
Solution Approach 1:
The system implements a universal control mechanism that handles both normal and recovery operational modes through a single gentle reduction factor parameter. This multi-functional approach allows the same control logic to adapt to different system states without requiring separate algorithms, maintaining simplicity while providing mode-specific behavior.
Data Source
AI summary
Techniques for producing a gentle reduction in throughput in a distributed service when a node of the service encounters a very large backlog of requests and/or when a previously offline node of the service is brought back online. These techniques may utilize multiple different algorithms to determine an amount of work that the distributed service is able to accept at any given time, rather than a single algorithm.