Distributed Storage Resource Scheduling for Node Failure Mis-Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face issues with excessive IO jitter and batch resource mis-scheduling due to storage node failures and network anomalies, affecting the stability and availability of cloud services.
Innovation Solution
A resource scheduling method that acquires request retry failure information to identify target storage resources and schedules them to healthy nodes, using a multi-level state machine and suppression strategies to minimize IO jitter and mis-scheduling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage node failure detection is performed using heartbeat mechanism, then system reliability is improved, but batch resource mis-scheduling occurs due to network anomalies
Solution Approach 1:
The system performs preliminary actions by acquiring request retry failure information before making scheduling decisions. When a storage node fails, the system first collects retry failure data from user sides, then uses this information to accurately identify which specific storage resources are affected, rather than immediately scheduling all resources on the failed node. This preliminary information gathering prevents premature and inaccurate scheduling decisions.
Solution Approach 2:
The system implements feedback mechanisms by continuously monitoring request retry failure information from user sides. When scheduling storage resources after a node failure, the system uses this feedback information to determine which specific resources need rescheduling. The feedback loop ensures that only resources that actually failed are rescheduled, avoiding batch mis-scheduling of resources that may have been unaffected by the failure or network anomalies.
2Stability of the object's composition
If all storage resources on a failed node are rescheduled, then system stability is improved, but IO jitter increases due to batch resource migration
Solution Approach 1:
The system extracts only the necessary storage resources for rescheduling by filtering based on request retry failure information. Instead of migrating all storage resources from a failed node, the system identifies and extracts only those specific resources that actually experienced failures. This selective extraction reduces the number of resources being migrated, thereby reducing IO jitter and avoiding unnecessary service interruptions for resources that were not affected.
Solution Approach 2:
The system applies local quality by treating different storage resources differently based on their actual failure status. Rather than applying a uniform rescheduling policy to all resources on a failed node, the system uses request retry failure information to identify which specific resources need rescheduling. This localized approach ensures that only affected resources undergo migration, minimizing overall IO jitter while maintaining stability for unaffected resources.
3Measurement precision
If request retry failure information is collected and analyzed, then scheduling precision is improved, but system complexity increases
Solution Approach 1:
The system implements self-service by having user sides automatically generate and report request retry failure information without requiring complex external monitoring infrastructure. The user sides themselves perform the measurement and reporting of failures, which simplifies the overall system architecture. The management node only needs to collect this self-reported information and use it for scheduling decisions, rather than implementing complex monitoring and detection mechanisms.
Data Source
AI summary
The present disclosure relates to a resource scheduling method and apparatus based on a distributed storage system, a medium, an electronic device, and a program product. The method includes: acquiring request retry failure information generated by a user side in a distributed storage cluster when a request retry fails, and storing the request retry failure information, wherein the request retry failure information is at least used to indicate a target storage resource on which the request retry fails; and in response to detecting that a target storage node in the distributed storage cluster has a heartbeat disconnection, scheduling, according to request retry failure information related to the target storage node, a target storage resource on which a request retry fails on the target storage node to another storage node in the distributed storage cluster that is in a normal operating state.