Cluster Maintenance Scheduling via Failure Domain Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current maintenance systems for computer clusters cause significant disruptions to clients due to the need to drain the entire cluster for maintenance, leading to inefficiencies and unpredictability in maintenance times.
Innovation Solution
A maintenance system that coordinates drain behavior among multiple clients, builds predictive models for drain duration, and schedules maintenance tasks to minimize disruption, allowing for more efficient and predictable machine drains by subdividing the cluster into failure domains and scheduling maintenance within existing resource constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire cluster is drained for maintenance, then all nodes can be maintained, but client service is completely disrupted
Solution Approach 1:
The patent divides the cluster into multiple failure domains (groups of machines with shared hardware dependencies). Maintenance is performed on one failure domain at a time while other domains continue to serve clients. This segmentation allows partial service availability during maintenance, resolving the contradiction between complete maintenance and continuous client service.
2Productivity
If drain time is reduced, then maintenance efficiency improves, but task eviction may exceed service level objectives
Solution Approach 1:
The patent implements dynamic drain time computation that adapts to current cluster state, job characteristics, and service level objectives. The system calculates optimal drain durations that satisfy SLOs while minimizing maintenance time. This dynamic adjustment resolves the contradiction between fast maintenance and SLO compliance by making drain time flexible rather than fixed.
Solution Approach 2:
The system changes multiple parameters simultaneously including drain duration, eviction rates, and task migration priorities to optimize maintenance while meeting SLOs. By adjusting these parameters dynamically based on cluster state and job characteristics, the system achieves both fast maintenance and SLO compliance.
3Adaptability or versatility
If multiple drains are scheduled independently, then maintenance flexibility increases, but resource contention increases and predictability decreases
Solution Approach 1:
The patent implements a feedback mechanism where the system monitors actual drain durations and uses this information to improve future drain time predictions. The feedback loop continuously refines the predictive model of drain behavior, enabling more accurate time estimates while maintaining scheduling flexibility. This resolves the contradiction between flexibility and predictability.
Solution Approach 2:
The system performs preliminary computation of drain times before scheduling drains, using predictive models to estimate durations based on current cluster state. This preliminary action allows the scheduler to make informed decisions about drain timing and resource allocation, improving both predictability and flexibility by knowing expected drain durations in advance.
Data Source
AI summary
A method of operating a cluster of machines includes receiving a request for a disruption, determining a subset of machines of the cluster affected by the requested disruption, and determining a set of jobs having corresponding tasks on the affected machines. The method also includes computing a drain time for a drain that drains the tasks of the jobs from the affected machines, and scheduling on a drain calendar stored in non-transitory memory a drain interval for the drain. The drain interval has a start time and an end time. A maintenance system that includes a scheduler may execute such a method to maintain a cluster of machines.