Backup Task Generation for Computer Cluster Job Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computer clusters, execution failures of scheduled jobs or tasks occur due to various reasons such as network failures, machine crashes, or overloaded resources, leading to de-scheduling risks that existing technologies struggle to effectively mitigate.
Innovation Solution
A system is implemented that creates backup tasks for scheduled jobs in computer clusters, classifying tasks based on resource requirements and determining the number of backup tasks needed to ensure a target probability of execution, with each backup task being identical and capable of replacing any scheduled task in its group, thereby minimizing de-scheduling risks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If backup tasks are created for all scheduled tasks, then job de-scheduling probability is reduced, but system complexity and resource overhead increase
Solution Approach 1:
The patent segments tasks into different groups based on their de-scheduling risk levels and resource requirements. Instead of creating backup tasks for all scheduled tasks uniformly, the system selectively creates backups only for high-risk tasks or tasks with specific characteristics. This segmentation approach reduces the overall number of backup tasks needed while maintaining reliability for critical jobs, thereby resolving the contradiction between improving reliability and reducing system complexity.
Solution Approach 2:
The patent applies different backup strategies to different task groups based on their specific characteristics. High-priority tasks or tasks with strict deadlines receive more robust backup protection, while lower-priority tasks use lighter backup mechanisms. This local quality differentiation ensures that reliability is improved where most needed without unnecessarily increasing system complexity across all tasks.
2Reliability
If more backup tasks are generated, then execution probability guarantee is improved, but resource consumption increases
Solution Approach 1:
The patent implements partial backup action by creating backup tasks only for a subset of scheduled tasks rather than all tasks. The system identifies tasks that require backup protection based on their execution probability targets and risk profiles, generating backups only for those tasks. This partial action approach ensures execution probability guarantees for critical tasks while avoiding the excessive resource consumption that would result from backing up every task in the system.
Solution Approach 2:
The patent dynamically adjusts the number and allocation of backup tasks based on changing system conditions and task priorities. When resources are abundant, more backup tasks may be created to enhance reliability. When resources are constrained, the system reduces backup creation while maintaining execution probability guarantees for high-priority tasks. This parameter adjustment allows the system to balance reliability guarantees with resource consumption based on current conditions.
3Productivity
If tasks are classified into multiple groups, then backup allocation efficiency is improved, but classification complexity increases
Solution Approach 1:
The patent segments tasks into distinct groups based on their resource requirements, priority levels, and execution characteristics. This segmentation enables the system to apply different backup allocation strategies to different groups, improving overall backup allocation efficiency. By categorizing tasks into manageable segments, the system can quickly determine which tasks need backups and how many backups are needed, rather than evaluating each task individually, thus resolving the contradiction between allocation efficiency and classification complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are provided for generating backup tasks (138) for a plurality of tasks (120) scheduled to run in a computer cluster (100). Each scheduled task is associated with a target probability for execution, and is executable by a first cluster element (102) and a second cluster element (104, 106). The system classifies the scheduled tasks into groups based on resource requirements of each task (602). The system determines the number of backup tasks to be generated. The number of backup tasks (528-532) is determined in a manner necessary to guarantee that the scheduled tasks satisfy the target probability for execution (800). The backup tasks are desirably identical for a given group. And each backup task can replace nay scheduled task in the given group.