Distributed Storage Resource Scheduling for Heartbeat Failure Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face issues with storage node failures causing excessive IO jitter and mis-scheduling due to heartbeat disconnections and network anomalies, leading to instability and prolonged recovery times.
Innovation Solution
A resource scheduling method that utilizes request retry failure information to identify and schedule storage resources to healthy nodes, employing a multi-level state machine and suppression strategies to minimize jitter and mis-scheduling, and includes pre-scheduling after heartbeat recovery to prevent secondary failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage node failure detection is performed using heartbeat mechanism, then system reliability is improved, but false mis-scheduling occurs due to network anomalies causing heartbeat disconnection
Solution Approach 1:
The system performs preliminary actions by collecting request retry failure information before making scheduling decisions. When a storage node's heartbeat disconnects, the system first checks whether there is actual request retry failure information indicating true failure, rather than immediately scheduling resources away. This preliminary verification prevents false mis-scheduling caused by transient network anomalies.
Solution Approach 2:
The system introduces feedback mechanisms by continuously monitoring request retry failure information from the user side and using this feedback to verify heartbeat disconnection events. The scheduling decision is based on feedback from both heartbeat status and request retry outcomes, ensuring that resource scheduling only occurs when there is confirmed evidence of storage node failure rather than temporary network issues.
2Stability of the object's composition
If resource scheduling is performed immediately upon heartbeat disconnection, then system stability is improved, but IO jitter increases due to unnecessary resource migration
Solution Approach 1:
Before performing resource scheduling, the system executes preliminary verification by checking request retry failure information. This preliminary action filters out false heartbeat disconnections caused by network anomalies, ensuring that resource migration only occurs when genuinely necessary. Consequently, unnecessary IO operations during resource migration are avoided, maintaining IO performance while preserving system stability.
Solution Approach 2:
The system applies local quality by differentiating between true storage node failures and false heartbeat disconnections. Instead of uniformly treating all heartbeat disconnections as failures, the system locally verifies each case using request retry failure information. This selective approach ensures resource scheduling is performed only where truly needed, avoiding unnecessary IO jitter in cases of transient network issues.
3Reliability
If all storage resources are migrated when a storage node fails, then system reliability is improved, but recovery time increases due to batch scheduling overhead
Solution Approach 1:
The system extracts and processes only the essential information needed for scheduling decisions - specifically request retry failure information related to the failed storage node. By focusing on extracting only the relevant failure data rather than processing all storage resources uniformly, the system reduces scheduling overhead and accelerates recovery time while maintaining reliability through targeted resource migration.
Solution Approach 2:
The system performs preliminary filtering of storage resources by identifying only those resources associated with request retry failures. This preliminary action allows the system to schedule only the necessary resources rather than migrating all resources from a failed node, thereby reducing batch scheduling overhead and shortening recovery time while preserving system reliability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates to a resource scheduling method and apparatus based on a distributed storage system, a medium, an electronic device and a program product. The method includes: acquiring request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and storing the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource on which the request retry fails; and in response to detecting that a target storage node in the distributed storage cluster has a heartbeat disconnection, scheduling, according to request retry failure information related to the target storage node, a target storage resource on which a request retry fails on the target storage node to another storage node in the distributed storage cluster that is in a normal operating state.